Symlink to directory in the build source confuses Garden CLI - it refuses to copy

[ohookins]

Bug

Current Behavior

We have a symlinked directory due to some internal node modules that needs to be copied (probably the symlink rather than the whole directory. For example:

./site/_assets/stylesheets/design-system -> ../../../packages/bc-flair/src/design-system

During a garden build it errors out:

Failed staging build for ds. Here is the output:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Attempted to copy non-file /full/path/to/repository/site/_assets/stylesheets/design-system
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
1 deploy task(s) failed!

And yet it's a symlink, not a non-file. Symlinks are supported by Garden CLI. Here's the relevant code:

https://github.com/garden-io/garden/blob/63b18bb27b2027bfaf1ed5e2ea4a4cb9a5af164d/core/src/build-staging/helpers.ts#L59-L74

In this case:

• the source is a symlink, but it’s pointing to a directory and not a file.
• the link has a target, so sourceStats is replaced with the target
• now sourceStats is the destination of the link, which is a directory
• line 68 checks to see if it’s a file, and it’s not, so it bails out

Expected behavior

The link itself should be copied to the build temporary directory.

Reproducible example

• Create a relative link to another directory in the source location for the build.
• garden build

Workaround

No real workaround for this - we need this symlink for our code to work.

Suggested solution(s)

The link itself should be copied to the build temporary directory.

Additional context

N/A

Your environment

• OS: MacOS 11.6.4
• How I'm running Kubernetes: EKS

garden version 0.12.37 😎

[thsig]

@edvald: I may be missing some context here, but would it be OK to simply change

if (!sourceStats.isFile()) {

to

if (!sourceStats.isFile() && !sourceStats.isDirectory()) {

here?

[edvald]

No, that won't do the trick unfortunately. Notice that this code is within cloneFile which is supposed to do just that. This needs to be handled further up. I can take a look.

[edvald]

I should note that link handling is generally a little tricky in these contexts. There's a reason, for example, that Docker doesn't allow links outside the build context, absolute links etc.

In this particular case, the question would of course arise: Should we translate the relative symlink to a corresponding relative path post-copy? What we try and do when possible is to traverse symlinks upon file scan, but when the links point outside the module root, we ignore them.

@ohookins could you try running with --verbose enabled and see if you see any symlink related warnings? (e.g. Ignoring symlink ...). Those warnings should probably be louder btw, making a side-note of that.

[edvald]

An additional concern with links pointing to an arbitrary destination is that we have no idea what's in that directory unless we traverse it, which would mean we have no idea if a build version should change. These matters are solvable but not easily solvable, so we should chat about what would make the most sense here.

[ohookins]

There's no --verbose so I've used log level 5, hopefully that's enough for you.

Error: 1 build task(s) failed!
    at Object.handleProcessResults (/snapshot/project/tmp/pkg/cli/node_modules/@garden-io/core/src/commands/base.ts:532:19)
    at BuildCommand.action (/snapshot/project/tmp/pkg/cli/node_modules/@garden-io/core/src/commands/build.ts:148:12)
    at GardenCli.runCommand (/snapshot/project/tmp/pkg/cli/node_modules/@garden-io/core/src/cli/cli.ts:498:20)
    at GardenCli.run (/snapshot/project/tmp/pkg/cli/node_modules/@garden-io/core/src/cli/cli.ts:660:26)
    at Object.runCli (/snapshot/project/tmp/pkg/cli/src/cli.ts:41:14)

Error Details:

results:
  stage-build.ds:
    type: stage-build
    description: staging build for ds
    key: stage-build.ds
    name: ds
    error:
      detail:
        from: >-
          /Users/ohookins/src/github.com/bugcrowd/ds/site/_assets/stylesheets/design-system
        to: >-
          /Users/ohookins/src/github.com/bugcrowd/ds/.garden/build/ds/site/_assets/stylesheets/design-system
        sourceStats:
          dev: 16777223
          mode: 16877
          nlink: 10
          uid: 502
          gid: 20
          rdev: 0
          blksize: 4096
          ino: 29865456
          size: 320
          blocks: 0
          atimeMs: 1648184524288.0942
          mtimeMs: 1648184520748.7954
          ctimeMs: 1648184520748.7954
          birthtimeMs: 1648184520706.8376
          atime: '2022-03-25T05:02:04.288Z'
          mtime: '2022-03-25T05:02:00.749Z'
          ctime: '2022-03-25T05:02:00.749Z'
          birthtime: '2022-03-25T05:02:00.707Z'
          path: >-
            /Users/ohookins/src/github.com/bugcrowd/ds/packages/bc-flair/src/design-system
      type: filesystem
    startedAt: '2022-06-22T05:01:41.450Z'
    completedAt: '2022-06-22T05:01:41.973Z'
    batchId: 271902a7-0761-410c-971e-e0cbc360161c
    version: v-fd5394de11

[xdvpser]

Having the same issue :( Is there any activity on it? @ohookins Did you find any workaround for this? Thanks in advance!

[vvagaytsev]

@xdvpser @ohookins sorry for the late reply. We're in the middle of a big refactoring now. We'll take a look after that refactoring is done. Most likely, after the Garden 0.13 release.

[stale[bot]]

This issue has been automatically marked as stale because it hasn't had any activity in 90 days. It will be closed in 14 days if no further activity occurs (e.g. changing labels, comments, commits, etc.). Please feel free to tag a maintainer and ask them to remove the label if you think it doesn't apply. Thank you for submitting this issue and helping make Garden a better product!

[vvagaytsev]

Closing as a duplicate of #2382