apko build --offline is insufficient

[anguslees]

https://kubernetes.slack.com/archives/C03MASRQP4M/p1728007778669529

Here's a bazel image build, note "--offline" is on the apko build command line:

SUBCOMMAND: # //some/path [action 'Action some/path', configuration: 01c762a3d178e643e0075307c3f73e8f366b0746649472a304719dc27336c29c, execution platform: //tools/build/bazel/remote_platform:engflow_rbe_x86_64, mnemonic: Action]
(cd <path> && \
  exec env - \
  /bin/bash -c 'cd bazel-out/k8-fastbuild-ST-9932050053a1/bin/some/path && ../apko_linux_amd64/apko $@' '' build some/path/apko.yaml step-ca-base:latest ../step-ca-base '--vcs=false' '--log-level=warn' '--include-paths=../' '--lockfile=../apko_images_zd90ae2ef/lockfile_copy' '--cache-dir=cache_step-ca-base' --offline --arch arm64)
# Configuration: 01c762a3d178e643e0075307c3f73e8f366b0746649472a304719dc27336c29c
# Execution platform: //tools/build/bazel/remote_platform:engflow_rbe_x86_64
INFO: Analyzed target //some/path:image (5 packages loaded, 2591 targets configured).
INFO: From Action some/path/step-ca-base:
2024/10/04 01:57:36 ERRO request failed error="Get \"https://packages.wolfi.dev/os/apk-configuration\": dial tcp: lookup packages.wolfi.dev on 169.254.169.254:53: dial udp 169.254.169.254:53: connect: network is unreachable" method=GET url=https://packages.wolfi.dev/os/apk-configuration
2024/10/04 01:57:37 ERRO request failed error="Get \"https://packages.wolfi.dev/os/apk-configuration\": dial tcp: lookup packages.wolfi.dev on 169.254.169.254:53: dial udp 169.254.169.254:53: connect: network is unreachable" method=GET url=https://packages.wolfi.dev/os/apk-configuration
2024/10/04 01:57:39 ERRO request failed error="Get \"https://packages.wolfi.dev/os/apk-configuration\": dial tcp: lookup packages.wolfi.dev on 169.254.169.254:53: dial udp 169.254.169.254:53: connect: network is unreachable" method=GET url=https://packages.wolfi.dev/os/apk-configuration
2024/10/04 01:57:43 ERRO request failed error="Get \"https://packages.wolfi.dev/os/apk-configuration\": dial tcp: lookup packages.wolfi.dev on 169.254.169.254:53: dial udp 169.254.169.254:53: connect: network is unreachable" method=GET url=https://packages.wolfi.dev/os/apk-configuration
2024/10/04 01:57:51 ERRO request failed error="Get \"https://packages.wolfi.dev/os/apk-configuration\": dial tcp: lookup packages.wolfi.dev on 169.254.169.254:53: dial udp 169.254.169.254:53: connect: network is unreachable" method=GET url=https://packages.wolfi.dev/os/apk-configuration
2024/10/04 01:57:51 WARN ignoring missing keys for https://packages.wolfi.dev/os: failed to perform key discovery: Get "https://packages.wolfi.dev/os/apk-configuration": GET https://packages.wolfi.dev/os/apk-configuration giving up after 5 attempt(s): Get "https://packages.wolfi.dev/os/apk-configuration": dial tcp: lookup packages.wolfi.dev on 169.254.169.254:53: dial udp 169.254.169.254:53: connect: network is unreachable

Note engflow's default execution sandbox (which I'm using here) doesn't even allow DNS lookups. The apko build succeeds, so this is just alarming noise afaict.

This was apko version:

GitVersion:    v0.19.2
GitCommit:     7844ca678c4b5b7408a54a12667d4c3e195056fd
GitTreeState:  clean
BuildDate:     '2024-09-20T20:06:22Z'
GoVersion:     go1.23.1
Compiler:      gc
Platform:      linux/amd64

@mattmoor suggests "Ah this is key discovery not being aware of offline."

[mattmoor]

@jonjohnsonjr I'm not familiar with how we implemented "offline". What's the best way to skip key discovery when in offline mode?

[sfc-gh-ptabor]

@mattmoor @jonjohnsonjr @imjasonh : Please take a look at the PR with proposed fix: https://github.com/chainguard-dev/apko/pull/1610

[krisharyan]

@anguslees @mattmoor can we close this issue

[sfc-gh-ptabor]

I think that most critical aspect of this issue is addressed: i.e. we don't get stuck for ~2-5min when running in offline mode...

Still there are some aspects pending to solve: a) I think apko is reaching out to Chainguards servers for index keys and auth even if in offline mode b) There are logs lines about fetching the keys (from URLs, but effectively from cache) and failing -> that are some redherring during the debugging.

[krisharyan]

Thank you for clarifying this Piotr

[anguslees]

The scary-looking error messages frequently mislead my colleagues when debugging unrelated build issues. I have to keep pointing them back to this bug report.

The scary messages are the issue here (functionality was always ok).

[xnox]

This was fixed on 15th April 2025 in:

• https://github.com/chainguard-dev/apko/releases/tag/v0.26.0

If there are any warnings printed, please check if you are using up to date apko and please open new bug reports.