Replaces of symlinks does not appear to work

[xnox]

with the modified wolfi-base as shown in https://github.com/wolfi-dev/os/pull/43832/files , which has two subpackages, that both ship the same symlink, to different destinations and with replaces declared of one another, one can install them with apk tools, but not with apko:

$ make package/wolfi-base
$ make local-wolfi

# apk add package-with-symlink
(1/1) Installing package-with-symlink (1-r8)
OK: 14 MiB in 16 packages
# readlink /etc/some.conf 
../run/target1.conf

# apk add package-with-symlink-replaces
(1/1) Installing package-with-symlink-replaces (1-r8)
OK: 14 MiB in 17 packages
# readlink /etc/some.conf 
../run/target2.conf
# exit



$ make test/wolfi-base
Testing package wolfi-base with version wolfi-base-1-r8 from file wolfi-base.yaml
/home/xnox/go/bin/melange test wolfi-base.yaml --repository-append /home/xnox/wolfi-dev/os/packages --keyring-append local-melange.rsa.pub --arch x86_64 --pipeline-dirs ./pipelines/ --repository-append https://packages.wolfi.dev/os --keyring-append https://packages.wolfi.dev/os/wolfi-signing.rsa.pub --test-package-append wolfi-base --debug  --source-dir ./wolfi-base/
2025/02/26 11:31:22 INFO building test workspace in: '/tmp/melange-guest-815115064-main' with apko
2025/02/26 11:31:22 [DEBUG] GET https://packages.wolfi.dev/os/apk-configuration
2025/02/26 11:31:22 INFO setting apk repositories: [/home/xnox/wolfi-dev/os/packages https://packages.wolfi.dev/os]
2025/02/26 11:31:22 INFO image configuration:
2025/02/26 11:31:22 INFO   contents:
2025/02/26 11:31:22 INFO     build repositories: []
2025/02/26 11:31:22 INFO     runtime repositories: []
2025/02/26 11:31:22 INFO     keyring:      []
2025/02/26 11:31:22 INFO     packages:     [package-with-symlink-replaces wolfi-base]
2025/02/26 11:31:22 INFO installing package-with-symlink (1-r8)
2025/02/26 11:31:22 INFO installing package-with-symlink-replaces (1-r8)
2025/02/26 11:31:22 INFO ERROR: failed to test package. the test environment has been preserved:
2025/02/26 11:31:22 INFO   workspace dir: /tmp/melange-workspace-2334097068
2025/02/26 11:31:22 INFO   guest dir: /tmp/melange-guest-815115064
2025/02/26 11:31:22 ERRO failed to test package: unable to build guest: unable to generate image: installing apk packages: installing packages: installing package-with-symlink-replaces (ver:1-r8 arch:x86_64): unable to install files for pkg package-with-symlink-replaces: unable to install symlink from etc/some.conf -> ../run/target2.conf: symlink ../run/target2.conf /tmp/melange-guest-815115064-main/etc/some.conf: file exists
make: *** [Makefile:115: test/wolfi-base] Error 1

Why is replaces not being honored by apko for symlinks, when apk-tools does? Note that replaces is honored by apko for regular files.

[xnox]

we should figure out what the replaces semantics should actually be

I guess we should check what happens with:

• directories
• regular files
• device nodes files
• symlinks
• hardlinks
• broken symlinks
• symlinks to directories
• directory to symlink replaces
• symlink to directory replaces
• etc.

[xnox]

Looking at apko code, writeHeader:

https://github.com/chainguard-dev/apko/blob/41f5c557366ee4ad9b70ddbf39fc89cf13fd2810/pkg/tarfs/fs.go#L437

implements replaces logic, but Symlink

https://github.com/chainguard-dev/apko/blob/41f5c557366ee4ad9b70ddbf39fc89cf13fd2810/pkg/tarfs/fs.go#L678

does not.

[iamrajiv]

@xnox I checked the WriteHeader function. I think the issue is that symlinks were taking a shortcut that bypassed the replaces logic.

        if hdr.Typeflag == tar.TypeSymlink {
            // TODO: I think we can drop this because the header checksum thing should catch this.
            if target, err := m.Readlink(hdr.Name); err == nil && target == hdr.Linkname {
                return false, nil
            }
        }

[xnox]

@iamrajiv yes they do => and the replaces logic higher up is not at all present there. And there is also not enough information passed through to this function. Thus likely need to keep track a little bit more information in the caller of this function; and then yeah, have replaces logic here too.

[iamrajiv]

Thanks @xnox, got it. Yeah, I can try to go through all the called functions, or what I was thinking is to make a sort of comprehensive test to verify the changes, so that we can understand what’s being missed here.

Also, the function lstat() should not follow symlinks, right? But I noticed it’s calling getNode(), which always follows symlinks.

[sergiodj]

Hi there,

I happened to stumble upon this issue (at least I believe it's the same issue!) today while trying to build gst-plugins-base:

2025/06/09 19:58:46 ERRO failed to build package: unable to build guest: unable to generate image: installing apk packages: installing packages: installing xorg-server (ver:21.1.16-r2 arch:x86_64): unable to install files for pkg xorg-server: error creating directory usr/share/X11/xkb: mkdir /home/user/tmp/melange-guest-3963903529/usr/share/X11/xkb: file exists

After some debugging and help from @jonjohnsonjr , we determined that the problem happens because xkeyboard-config creates a symlink from /usr/share/X11/xkb to /usr/share/xkeyboard-config-2, while xorg-server has a directory name /usr/share/X11/xkb.

According to:

https://github.com/chainguard-dev/apko/blob/296c34d294a984f196150f18d89a33320b676c54/pkg/apk/apk/install.go#L218-L228

this is a case which should be properly handled by apko, but apparently it isn't.

Further investigative work led to a few questions/conclusions:

• It seems like our Lstat implementation doesn't return proper information about symlinks?

https://github.com/chainguard-dev/apko/blob/296c34d294a984f196150f18d89a33320b676c54/pkg/apk/fs/memfs.go#L156-L168

• For that reason, the if fi, err := a.fs.Stat(header.Name); err == nil && fi.Mode()&os.ModeSymlink != 0 check (from installAPKFiles) never succeeds.

[iamrajiv]

Thanks for the insights @xnox

Okay so what I understand is that this issue is broader than it initially appeared. As of now we have identified three issues. Please correct me if I am wrong

1. Symlink Replaces Not Working When APK packages with "replaces" declarations try to install symlinks that conflict with existing symlinks, apko fails with a "file exists" error, whereas apk tools handles this correctly

2. Directory vs Symlink Conflict Issue (gst plugins base) There is a build failure when one package creates a symlink and another tries to create a directory at the same path

3. Fundamental Lstat Implementation Problem Both pkg/apk/fs/memfs.go and pkg/tarfs/fs.go have broken Lstat implementations

Out of these, for the first issue, I found that pkg/tarfs/fs.go has an early return for symlinks that bypasses the replaces logic. Symlinks take a shortcut path and never reach writeHeader where the replaces logic is implemented. I can try fixing this by removing the early return so that symlinks go through the same replaces logic as regular files first and then tackle the other issues later.

wdyt?