Scaffolder does not properly support broken symlinks end to end

[marcus-crane]

A continuation of the discussion in #12377, working through getting the scaffolder to support broken symlinks.

In particular, symlinks that will resolve in their destination repo but are not technically valid in the source template folder or when in scaffolder memory

Expected Behavior

When a symlink is part of a template, it should be properly committed and pushed to a destination repository

Actual Behavior

The symlink makes its way through the scaffolder pipeline but it is not included in the resulting pull request.

Steps to Reproduce

1. Create a scaffolder template with a "broken" symlink as one of the source files to be templated
2. Attempt to open a PR containing the symlink
3. Notice that the resulting pull request is missing the symlink within the template

Context

I've managed to pinpoint the exact cause which is the serializeDirectoryContents function within the scaffolder-backend

Specifically, when globby is invoked to pick up files in the source directory, it doesn't properly obtain symlinks.

Here's a minimal replication of the relevant portion of serializeDirectoryContents applied to a folder containing an empty text file and a broken symlink.

We can see above that only the text file is picked up.

However, if we add onlyFiles: false to the Globby configuration, as suggested here, we can see that both files are picked up as we might expect.

This fix would not entirely symlink support end to end however as the next step will try to read the contents of the symlink which will fail.

The use of onlyFiles would also pick up directories but those can be filtered out of the pipeline before files are parsed by making use of fs.dirent which is already available thanks to Globby's use of objectmode

Given our particular use case, I can try and craft a PR that has this working and submit it once I've got it working end to end

Your Environment

• Browser Information: Arc Version 0.62.0 (33900) Chromium Engine Version 104.0.5112.81

• Output of yarn backstage-cli info:

$ /Users/marcus/Code/work/backstage/node_modules/.bin/backstage-cli info
OS:   Darwin 22.0.0 - darwin/x64
node: v16.14.0
yarn: 1.22.19
cli:  0.17.2 (installed)

Dependencies:
  @backstage/app-defaults                          1.0.3
  @backstage/backend-common                        0.14.0
  @backstage/backend-tasks                         0.3.2
  @backstage/backend-test-utils                    0.1.25
  @backstage/catalog-client                        1.0.3
  @backstage/catalog-model                         1.0.3
  @backstage/cli-common                            0.1.9
  @backstage/cli                                   0.17.2
  @backstage/config-loader                         1.1.2
  @backstage/config                                1.0.1
  @backstage/core-app-api                          1.0.3
  @backstage/core-components                       0.9.5
  @backstage/core-plugin-api                       1.0.3
  @backstage/errors                                1.0.0
  @backstage/integration-react                     1.1.1
  @backstage/integration                           1.2.1
  @backstage/plugin-api-docs                       0.8.6
  @backstage/plugin-app-backend                    0.3.33
  @backstage/plugin-auth-backend                   0.14.1
  @backstage/plugin-auth-node                      0.2.2
  @backstage/plugin-bazaar-backend                 0.1.17
  @backstage/plugin-bazaar                         0.1.21
  @backstage/plugin-catalog-backend-module-aws     0.1.6
  @backstage/plugin-catalog-backend-module-github  0.1.4
  @backstage/plugin-catalog-backend                1.2.0
  @backstage/plugin-catalog-common                 1.0.3
  @backstage/plugin-catalog-graph                  0.2.18
  @backstage/plugin-catalog-import                 0.8.9
  @backstage/plugin-catalog-react                  1.1.1
  @backstage/plugin-catalog                        1.3.0
  @backstage/plugin-cicd-statistics                0.1.8
  @backstage/plugin-circleci                       0.3.6
  @backstage/plugin-cost-insights                  0.11.28
  @backstage/plugin-explore-react                  0.0.18
  @backstage/plugin-explore                        0.3.37
  @backstage/plugin-github-actions                 0.5.6
  @backstage/plugin-github-pull-requests-board     0.1.0
  @backstage/plugin-home                           0.4.22
  @backstage/plugin-kafka-backend                  0.2.26
  @backstage/plugin-kafka                          0.3.6
  @backstage/plugin-org                            0.5.6
  @backstage/plugin-pagerduty                      0.4.0
  @backstage/plugin-permission-common              0.6.2
  @backstage/plugin-permission-node                0.6.2
  @backstage/plugin-permission-react               0.4.2
  @backstage/plugin-proxy-backend                  0.2.27
  @backstage/plugin-scaffolder-backend             1.3.1
  @backstage/plugin-scaffolder-common              1.1.1
  @backstage/plugin-scaffolder                     1.3.0
  @backstage/plugin-search-backend-module-pg       0.3.4
  @backstage/plugin-search-backend-node            0.6.2
  @backstage/plugin-search-backend                 0.5.3
  @backstage/plugin-search-common                  0.3.5
  @backstage/plugin-search-react                   0.2.1
  @backstage/plugin-search                         0.9.0
  @backstage/plugin-stack-overflow                 0.1.2
  @backstage/plugin-tech-radar                     0.5.13
  @backstage/plugin-techdocs-backend               1.1.2
  @backstage/plugin-techdocs-module-addons-contrib 1.0.1
  @backstage/plugin-techdocs-node                  1.1.2
  @backstage/plugin-techdocs-react                 1.0.1
  @backstage/plugin-techdocs                       1.2.0
  @backstage/plugin-user-settings                  0.4.5
  @backstage/release-manifests                     0.0.4
  @backstage/test-utils                            1.1.1
  @backstage/theme                                 0.2.15
  @backstage/types                                 1.0.0
  @backstage/version-bridge                        1.0.1

[marcus-crane]

cc @pvandervelde

[marcus-crane]

I've got a minimum viable fix for this that I'll submit as a draft tomorrow 🙂