stow command is failing in UDI build

[svor]

Describe the bug

Stow command is failing with the error:

#56 6.026 All operations aborted.
#56 ERROR: process "/bin/sh -c stow . -t /home/user/ -d /home/tooling/ --no-folding" did not complete successfully: exit code: 1
------
 > [50/53] RUN stow . -t /home/user/ -d /home/tooling/ --no-folding:
6.026 WARNING! stowing . would cause conflicts:
6.026   * source is an absolute symlink ../tooling/.krew/bin/kubectl-ctx => /home/tooling/.krew/store/ctx/v0.9.5/kubectx
6.026   * source is an absolute symlink ../tooling/.krew/bin/kubectl-krew => /home/tooling/.krew/store/krew/v0.4.4/krew
6.026   * source is an absolute symlink ../tooling/.krew/bin/kubectl-ns => /home/tooling/.krew/store/ns/v0.9.5/kubens
6.026   * source is an absolute symlink ../tooling/.sdkman/candidates/java/current => /home/tooling/.sdkman/candidates/java/17.0.10-tem
6.026 All operations aborted.

It's reproducible locally and in the PR check: https://github.com/devfile/developer-images/actions/runs/8984086811/job/24717150614?pr=171

Che version

next (development version)

Steps to reproduce

Try to build UDI https://github.com/devfile/developer-images/

Expected behavior

UDI should be built without errors

Runtime

OpenShift

Screenshots

No response

Installation method

chectl/next

Environment

Linux

Eclipse Che Logs

No response

Additional context

No response

[svor]

@AObuchow could you take a look please

[AObuchow]

This has started happening because stow 2.4.0 was recently released and pushed to the epel (Extra Packages for Enterprise Linux 8 - x86_64) repos. I still haven't isolated the exact stow commit which resulted in this change, but it seems that stow doesn't officially support stowing absolute symbolic links. I believe there was a bug in stow that previously allowed stowing absolute symbolic links in certain cases (maybe if they were in a directory starting with a .?), which has been resolved in stow 2.4.0.

I still need to figure out how to resolve this. It doesn't seem like we can simply have dnf install the previous version of stow (2.3.1) since it doesn't seem to be available in the epel 8 repo anymore.

We could potentially:

• Downgrade by curl-ing stow to install it from a GNU mirror (as suggested on the "Downloading Stow" section of stow's official page. For the downstream UDI where we are manually building stow, we'd simply checkout the v2.3.1 tag from the repository. We might have to take this approach as a temporary workaround.
• Ignore stowing files in /home/tooling/ which we know are absolute symbolic links (this seems messy and difficult to maintain)

I'd like to figure out a proper way to handle these absolute symbolic links with stow, but need to investigate further.

[AObuchow]

Note: if we go with the fix of downgrading stow to 2.3.1 as done here, we'll have to make a similar change for the downstream UDI.

[AObuchow]

Update: My fix to downgrade stow has a bug in it where the /usr/share/ files used by stow are not properly copied when installing stow 2.3.1. As a result, stow is still currently broken in the UDI.

I've decided to re-open this issue (to simplify tracking the order of events that happened here) and will instead opt for using the latest version of stow 2.4.0 and adding the absolute symbolic links present in /home/tooling/ to the .stow-local-ignore. This is the solution that a DevSpaces customer has taken, and I would prefer if we mirrored their solution so that we can catch any potential issues with this solution (though, I am hopeful none should arise as long as /home/tooling/... remains on $PATH).

Edit: the latest PR for resolving this issue is https://github.com/devfile/developer-images/pull/173