apko icon indicating copy to clipboard operation
apko copied to clipboard
verified publisher icon chainguard-dev

local packages not picked up by apko

Open Lunarequest opened this issue 2 years ago • 24 comments

I used melange to build a apk and it created a local package repo. I used the following command to do this

docker run --rm --privileged -v "${PWD}":/work \                                                                                                                               
    cgr.dev/chainguard/melange build melange.yaml \
    --arch amd64,aarch64 \
    --repository-append packages --signing-key melange.rsa

I tried to use that to try and build an oci image using the following apko.yaml

contents:
  keyring:
    - https://packages.wolfi.dev/os/wolfi-signing.rsa.pub
    - melange.rsa.pub
  repositories:
    - https://packages.wolfi.dev/os
    - '@local /work/packages'
  packages:
    - ca-certificates-bundle
    - wolfi-base
    - Lunas-blog@local
accounts:
  groups:
    - groupname: nonroot
      gid: 65532
  users:
    - username: nonroot
      uid: 65532
  run-as: 65532
entrypoint:
  command: /usr/bin/blog

I used this command to generate the oci image

docker run -v "$PWD":/work cgr.dev/chainguard/apko build --arch amd64,aarch64 apko.yaml blog:latest blog.tar

I originally thought this may have been an issue with podman so I installed docker and tried to use it instead, there was no effect so I'm led to believe this is an issue with apko

Lunarequest avatar Mar 08 '23 01:03 Lunarequest

Have you updated the apko image? What does apko version say?

kaniini avatar Mar 08 '23 03:03 kaniini

I have updated the image and apko version says

     _      ____    _  __   ___
    / \    |  _ \  | |/ /  / _ \
   / _ \   | |_) | | ' /  | | | |
  / ___ \  |  __/  | . \  | |_| |
 /_/   \_\ |_|     |_|\_\  \___/
apko

GitVersion:    v0.7.1
GitCommit:     unknown
GitTreeState:  unknown
BuildDate:     unknown
GoVersion:     go1.20.1
Compiler:      gc
Platform:      linux/amd64

Lunarequest avatar Mar 08 '23 05:03 Lunarequest

I've got the same problem the error says even if I specify the local packages as @local /work/packages:

Error: failed to build layer image: installing apk packages: error getting package dependencies: error getting repository indexes: failed to read repository @local/x86_64/APKINDEX.tar.gz: open @local/x86_64/APKINDEX.tar.gz: no such file or directory
2023/03/08 14:07:59 error during command execution: failed to build layer image: installing apk packages: error getting package dependencies: error getting repository indexes: failed to read repository @local/x86_64/APKINDEX.tar.gz: open @local/x86_64/APKINDEX.tar.gz: no such file or directory

developer-guy avatar Mar 08 '23 14:03 developer-guy

@Lunarequest I found a workaround with something like this:

contents:
  keyring:
    - https://packages.wolfi.dev/os/wolfi-signing.rsa.pub
    - melange.rsa.pub
  repositories:
    - https://packages.wolfi.dev/os
    - /work/packages # List just a reference to the directory
  packages:
    - ca-certificates-bundle
    - wolfi-base
    - Lunas-blog # No @local postFix
accounts:

satdeveloping avatar Mar 08 '23 22:03 satdeveloping

I can confirm the above works ^

Lunarequest avatar Mar 09 '23 01:03 Lunarequest

yep, @strongjz also recommended using the --keyring-append and --repository-append flags which both do the same thing above ^

developer-guy avatar Mar 09 '23 09:03 developer-guy

@deitch is masked packages working again in our implementation?

kaniini avatar Mar 16 '23 20:03 kaniini

There was #539 which added it, and then #550 which fixed a single error. It has worked every time I tried it.

If we have a reproduction of an error, I am more than happy to tackle it.

deitch avatar Mar 17 '23 12:03 deitch

Can we close this? I just ran another test, using the below apko.yaml, worked like a charm:

contents:
  keyring:
    - https://packages.wolfi.dev/os/wolfi-signing.rsa.pub
  repositories:
    - "@local /tmp/apk"
  packages:
    - glibc@local
    - wolfi-baselayout@local

I copied down the glibc, glibc-locale-posix and wolfi-baselayout apk files to /tmp/apk.

deitch avatar Mar 23 '23 11:03 deitch

I pulled the latest cgr.dev/chainguard/melange with this apko.yaml modified with this patch

diff --git a/apko.yaml b/apko.yaml
index 913826a..743465e 100644
--- a/apko.yaml
+++ b/apko.yaml
@@ -4,11 +4,11 @@ contents:
     - melange.rsa.pub
   repositories:
     - https://packages.wolfi.dev/os
-    - '/work/packages'
+    - "@local /work/packages"
   packages:
     - ca-certificates-bundle
     - wolfi-baselayout
-    - Lunas-blog
+    - Lunas-blog@local
 accounts:
   groups:
     - groupname: nonroot

and I still get this error

Error: failed to build layer image for "amd64": installing apk packages: error getting package dependencies: error getting repository indexes: failed to read repository @local/x86_64/APKINDEX.tar.gz: open @local/x86_64/APKINDEX.tar.gz: no such file or directory
2023/03/23 11:18:27 error during command execution: failed to build layer image for "amd64": installing apk packages: error getting package dependencies: error getting repository indexes: failed to read repository @local/x86_64/APKINDEX.tar.gz: open @local/x86_64/APKINDEX.tar.gz: no such file or directory

Lunarequest avatar Mar 23 '23 11:03 Lunarequest

cgr.dev/chainguard/melange

Did you mean cgr.dev/chainguard/apko ?

deitch avatar Mar 23 '23 11:03 deitch

cgr.dev/chainguard/melange

Did you mean cgr.dev/chainguard/apko ?

yes sorry I copy pasted the wrong thing

Lunarequest avatar Mar 23 '23 11:03 Lunarequest

Can you try the latest explicitly more recently? cgr.dev/chainguard/apko:latest-20230322

Not 100% convinced that is the issue, but maybe it is a tagging issue; let's rule it out.

deitch avatar Mar 23 '23 11:03 deitch

that's the exact command and the output

docker run --rm -v "${PWD}":/work \                                                                                                                                           
    -e REF="${REF}" \
    -e GITHUB_USERNAME="${GITHUB_USERNAME}" \
    -e GITHUB_TOKEN="${GITHUB_TOKEN}" \
    --entrypoint sh \
    cgr.dev/chainguard/apko:latest-20230322  -c \
        'echo "${GITHUB_TOKEN}" | \         
            apko login ghcr.io -u "${GITHUB_USERNAME}" --password-stdin && \
            apko publish --debug apko.yaml \
                "${REF}" -k melange.rsa.pub \
                --arch amd64'
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
Trying to pull cgr.dev/chainguard/apko:latest-20230322...
Getting image source signatures
Copying blob 53a8597c4d4e done  
Copying config e7ae5c2612 done  
Writing manifest to image destination
Storing signatures
2023/03/23 11:34:58 logged in via /root/.docker/config.json
Mar 23 11:34:58.102 [INFO] loading config file: apko.yaml
Mar 23 11:34:58.102 [INFO] [arch:x86_64] detected git+ssh://github.com/Lunarequest/blog-backend.git@17193cd50139c13e1031224fcd10415039bdef86 as VCS URL
Mar 23 11:34:58.102 [INFO] [arch:x86_64] Publishing images for 1 architectures: [amd64]
Mar 23 11:34:58.102 [INFO] [arch:x86_64] building tags [ghcr.io/lunarequest/blog:latest]
Mar 23 11:34:58.102 [INFO] loading config file: apko.yaml
Mar 23 11:34:58.102 [INFO] [arch:x86_64] detected git+ssh://github.com/Lunarequest/blog-backend.git@17193cd50139c13e1031224fcd10415039bdef86 as VCS URL
Mar 23 11:34:58.103 [INFO] [arch:x86_64] build context:
Mar 23 11:34:58.103 [INFO] [arch:x86_64]   working directory: /tmp/apko-3326349783/x86_64
Mar 23 11:34:58.103 [INFO] [arch:x86_64]   tarball path: 
Mar 23 11:34:58.103 [INFO] [arch:x86_64]   source date: 1970-01-01 00:00:00 +0000 UTC
Mar 23 11:34:58.103 [INFO] [arch:x86_64]   Docker mediatypes: false
Mar 23 11:34:58.103 [INFO] [arch:x86_64]   SBOM output path: 
Mar 23 11:34:58.103 [INFO] [arch:x86_64]   arch: x86_64
Mar 23 11:34:58.103 [INFO] [arch:x86_64] image configuration:
Mar 23 11:34:58.103 [INFO] [arch:x86_64]   contents:
Mar 23 11:34:58.103 [INFO] [arch:x86_64]     repositories: [https://packages.wolfi.dev/os @local /work/packages]
Mar 23 11:34:58.103 [INFO] [arch:x86_64]     keyring:      [https://packages.wolfi.dev/os/wolfi-signing.rsa.pub melange.rsa.pub]
Mar 23 11:34:58.103 [INFO] [arch:x86_64]     packages:     [ca-certificates-bundle wolfi-baselayout Lunas-blog@local]
Mar 23 11:34:58.103 [INFO] [arch:x86_64]   entrypoint:
Mar 23 11:34:58.103 [INFO] [arch:x86_64]     type:    
Mar 23 11:34:58.103 [INFO] [arch:x86_64]     command:     /usr/bin/blog
Mar 23 11:34:58.103 [INFO] [arch:x86_64]     service: map[]
Mar 23 11:34:58.103 [INFO] [arch:x86_64]     shell fragment: 
Mar 23 11:34:58.103 [INFO] [arch:x86_64]   accounts:
Mar 23 11:34:58.103 [INFO] [arch:x86_64]     runas:  65532
Mar 23 11:34:58.103 [INFO] [arch:x86_64]     users:
Mar 23 11:34:58.103 [INFO] [arch:x86_64]       - uid=65532(nonroot) gid=0
Mar 23 11:34:58.103 [INFO] [arch:x86_64]     groups:
Mar 23 11:34:58.103 [INFO] [arch:x86_64]       - gid=65532(nonroot) members=[]
Mar 23 11:34:58.103 [INFO] [arch:x86_64] doing pre-flight checks
Mar 23 11:34:58.103 [INFO] [arch:x86_64] building image fileystem in /tmp/apko-3326349783/x86_64
Mar 23 11:34:58.103 [INFO] [arch:x86_64] initializing apk database
Mar 23 11:34:59.622 [INFO] [arch:x86_64] finished initializing apk database
Mar 23 11:34:59.622 [INFO] [arch:x86_64] setting apk world
Mar 23 11:34:59.623 [INFO] [arch:x86_64] initializing apk keyring
Mar 23 11:34:59.623 [INFO] [arch:x86_64] setting apk repositories
Mar 23 11:34:59.623 [DEBUG] [arch:x86_64] appending 1 extra keys to keyring
Mar 23 11:34:59.623 [DEBUG] [arch:x86_64] installing key melange.rsa.pub
Mar 23 11:34:59.623 [DEBUG] [arch:x86_64] installing key https://packages.wolfi.dev/os/wolfi-signing.rsa.pub
Mar 23 11:34:59.623 [DEBUG] [arch:x86_64] installing key melange.rsa.pub
Mar 23 11:34:59.854 [INFO] [arch:x86_64] synchronizing with desired apk world
Mar 23 11:34:59.854 [INFO] [arch:x86_64] determining desired apk world
Error: failed to build layer image for "amd64": installing apk packages: error getting package dependencies: error getting repository indexes: failed to read repository @local/x86_64/APKINDEX.tar.gz: open @local/x86_64/APKINDEX.tar.gz: no such file or directory
2023/03/23 11:34:59 error during command execution: failed to build layer image for "amd64": installing apk packages: error getting package dependencies: error getting repository indexes: failed to read repository @local/x86_64/APKINDEX.tar.gz: open @local/x86_64/APKINDEX.tar.gz: no such file or directory

Lunarequest avatar Mar 23 '23 11:03 Lunarequest

OK @Lunarequest I can reproduce it now using the image, although a locally built version fails. It would help if I could see the commit on your version, but there is something wrong with the version output, see #591

In any case, your issue is fixed, but for some reason is not picked up in the published images. Let's hunt that down.

Thanks for the patience in figuring this out.

deitch avatar Mar 23 '23 11:03 deitch

Good suggestion from @kaniini , can you try this image? ghcr.io/wolfi-dev/sdk:latest

deitch avatar Mar 23 '23 11:03 deitch

Good suggestion from @kaniini , can you try this image? ghcr.io/wolfi-dev/sdk:latest

how would i use it?

Lunarequest avatar Mar 23 '23 12:03 Lunarequest

Oh sorry. Forgot that it is the whole big sdk, not just apko.

This is what I do:

$ docker run -it --rm -v /path/to/apko.yaml:/tmp/apko.yaml:ro ghcr.io/wolfi-dev/sdk:latest

Welcome to the development environment!

[sdk] ❯ apko build /tmp/apko.yaml apko /tmp/output.tar

deitch avatar Mar 23 '23 12:03 deitch

You can use it the same way you are using the apko image.

kaniini avatar Mar 23 '23 12:03 kaniini

I've got the same problem. I try that with the sdk image:

[sdk] ❯ apko build work/apko.yaml apko /tmp/output.tar -k work/melange.rsa.pub --arch host
ℹ            | loading config file: work/apko.yaml
ℹ  aarch64   | Building images for 1 architectures: [arm64]
ℹ  aarch64   | building tags [apko]
ℹ            | loading config file: work/apko.yaml
ℹ  aarch64   | detected git+ssh://github.com/portefaix/portefaix-distroless.git@a161a1b4d7c3f5e64f65184c625410d5f9a924cf as VCS URL
ℹ  aarch64   | doing pre-flight checks
ℹ  aarch64   | building apk info in /tmp/apko-2619998972/aarch64
ℹ  aarch64   | initializing apk database
2023/08/18 07:35:23 [DEBUG] GET https://alpinelinux.org/releases.json
2023/08/18 07:35:23 [DEBUG] GET https://alpinelinux.org/keys/alpine-devel%40lists.alpinelinux.org-616ae350.rsa.pub
2023/08/18 07:35:23 [DEBUG] GET https://alpinelinux.org/keys/alpine-devel%40lists.alpinelinux.org-616ae350.rsa.pub
ℹ  aarch64   | finished initializing apk database
ℹ  aarch64   | initializing apk keyring
ℹ  aarch64   | setting apk world
ℹ  aarch64   | setting apk repositories
ℹ  aarch64   | build context:
ℹ  aarch64   | build options:
{
        "withVCS": true,
        "workDir": "/tmp/apko-2619998972/aarch64",
        "tags": [
                "apko"
        ],
        "sourceDateEpoch": "1970-01-01T00:00:00Z",
        "sbomPath": "/tmp/apko-2619998972/image",
        "sbomFormats": [
                "spdx",
                "cyclonedx"
        ],
        "extraKeyFiles": [
                "work/melange.rsa.pub"
        ],
        "arch": "arm64",
        "Log": {
                "Out": {},
                "Fields": null,
                "Level": 4
        }
}
ℹ  aarch64   | image configuration:
ℹ  aarch64   |   contents:
ℹ  aarch64   |     repositories: [https://dl-cdn.alpinelinux.org/alpine/edge/main https://dl-cdn.alpinelinux.org/alpine/edge/community @local /work/packages]
ℹ  aarch64   |     keyring:      []
ℹ  aarch64   |     packages:     [alpine-baselayout-data ca-certificates-bundle tzdata busybox curl wget portefaix-distroless@local]
ℹ  aarch64   |   entrypoint:
ℹ  aarch64   |     type:
ℹ  aarch64   |     command:     /usr/bin/entrypoint.sh
ℹ  aarch64   |     service: map[]
ℹ  aarch64   |     shell fragment:
ℹ  aarch64   |   accounts:
ℹ  aarch64   |     runas:  65532
ℹ  aarch64   |     users:
ℹ  aarch64   |       - uid=65532(nonroot) gid=65532
ℹ  aarch64   |     groups:
ℹ  aarch64   |       - gid=65532(nonroot) members=[]
ℹ  aarch64   |     annotations:
ℹ  aarch64   |       org.opencontainers.image.source: https://github.com/portefaix/portefaix-distroless/
ℹ  aarch64   |       org.opencontainers.image.documentation: https://docs.portefaix.xyz/
ℹ  aarch64   |       org.opencontainers.image.url: https://portefaix.xyz
ℹ  aarch64   | synchronizing with desired apk world
ℹ  aarch64   | determining desired apk world
2023/08/18 07:35:23 [DEBUG] HEAD https://dl-cdn.alpinelinux.org/alpine/edge/community/aarch64/APKINDEX.tar.gz
2023/08/18 07:35:24 [DEBUG] HEAD https://dl-cdn.alpinelinux.org/alpine/edge/main/aarch64/APKINDEX.tar.gz
ℹ  aarch64   | creating group 65532(nonroot)
⚠  aarch64   | did not find /etc/os-release at etc/os-release
⚠  aarch64   | distro ID not specified and /etc/os-release does not already exist
ℹ  aarch64   | generating supervision tree
ℹ  aarch64   | finished building filesystem in /tmp/apko-2619998972/aarch64
ℹ  aarch64   | built image layer tarball as /tmp/apko-temp-3961776234/apko-aarch64.tar.gz
ℹ  aarch64   | building image from layer
ℹ  aarch64   | OCI layer digest: sha256:b9d541b40cf1b933734184b09bae78b1a3f6ac5c9f56f044c1e90550c37ef54e
ℹ  aarch64   | OCI layer diffID: sha256:f52ecb904ede1a12eca158ce479aceb08feff4a24eb5b327fbe4d0f13efc223d
ℹ            | loading config file: work/apko.yaml
ℹ  aarch64   | detected git+ssh://github.com/portefaix/portefaix-distroless.git@a161a1b4d7c3f5e64f65184c625410d5f9a924cf as VCS URL
ℹ  aarch64   | doing pre-flight checks
ℹ  aarch64   | building apk info in /tmp/apko-2619998972
ℹ  aarch64   | initializing apk database
2023/08/18 07:35:25 [DEBUG] GET https://alpinelinux.org/releases.json
2023/08/18 07:35:25 [DEBUG] GET https://alpinelinux.org/keys/alpine-devel%40lists.alpinelinux.org-616ae350.rsa.pub
2023/08/18 07:35:25 [DEBUG] GET https://alpinelinux.org/keys/alpine-devel%40lists.alpinelinux.org-616ae350.rsa.pub
ℹ  aarch64   | finished initializing apk database
ℹ  aarch64   | initializing apk keyring
ℹ  aarch64   | setting apk repositories
ℹ  aarch64   | setting apk world
ℹ  aarch64   | built index file as /tmp/apko-temp-2343703950/index.json
INFO[0002] Generating arch image SBOMs
ℹ  aarch64   | Generating image SBOM for arm64
ℹ  aarch64   | Generating index SBOM
INFO[0002] Final index tgz at: /tmp/output.tar

nlamirault avatar Aug 18 '23 07:08 nlamirault

how can we fix this ?

nlamirault avatar Sep 06 '23 06:09 nlamirault

As far as I can tell the @local syntax (or any pinning syntax) no longer works (for example, the config as defined in https://github.com/chainguard-dev/nginx-image-demo/) no longer installs the correct version of nginx when it's run.

The code referenced in https://github.com/chainguard-dev/apko/issues/557#issuecomment-1473758997 also no longer appears to exist either.

As others have stated above, the only solution I have found is to avoid using the pinning syntax entirely, and reference the necessary repositories directly. The ordering of the repositories also appears to matter if you have a package which is available by the same name in multiple repositories (packaging is not my area of expertise!).

ferozsalam avatar Mar 04 '24 17:03 ferozsalam

Unfortunately, I have not found documentation about this.

So, if someone wants to do something similar, here are two working examples:

Hopefully, it may help someone else.

hpedrorodrigues avatar Jul 14 '24 03:07 hpedrorodrigues

For those having this issue (which isn't being picked up by the maintainers, so has unclear status). Validate the contents of APKINDEX.tar.gz. Running Melange multiple times separately, as part of CI for example, may clobber this index file to contain on the last built package. Then Apko will not find other packages you're installing out of this index, even if the package files themselves are there.

sanmai-NL avatar Sep 09 '25 07:09 sanmai-NL