`githubRelease` errors on releases without explicit assets

[LukeWinikates]

What steps did you take:

I attempted to run vendir sync with a githubRelease for open-policy-agent/gatekeeper. The entry under contents: looks like:

  - path: gatekeeper
    githubRelease:
      slug: open-policy-agent/gatekeeper
      latest: true
      disableAutoChecksumValidation: true

What happened: I get an error that reads Error: Syncing directory 'config/upstream': Filtering paths in directory 'gatekeeper': Expected to find at least one file within directory. This error is very cryptic. I spent a few troubleshooting cycles creating empty files in config/upstream/gatekeeper or deleting the directory, thinking that the error message was complaining about the local filesystem.

Reading the vendir code, the problem seems to be that there’s no particular asset tied to the release - this project doesn’t provide binaries with their releases. You can download the source code, but that’s not listed in the github API json response as part of the assets array.

the release in question: https://api.github.com/repos/open-policy-agent/gatekeeper/releases/37006810

...
  "assets": [

  ],
  "tarball_url": "https://api.github.com/repos/open-policy-agent/gatekeeper/tarball/v3.3.0",
  "zipball_url": "https://api.github.com/repos/open-policy-agent/gatekeeper/zipball/v3.3.0",
...

https://github.com/vmware-tanzu/carvel-vendir/blob/1da965479755f7b3583e275d337e66f6d7a25950/pkg/vendir/fetch/githubrelease/sync.go#L76 refers specifically to the assets array, and loops over that array, looking for assets to copy. Finding none, it errors out, apparently at https://github.com/vmware-tanzu/carvel-vendir/blob/bf0685a51eae291131fe5d1475ac0a6eceeb3c1e/pkg/vendir/directory/directory.go#L160

What did you expect: I expected to be able to download the source code, since the Github UI shows it as an available download, and only after looking at the implementation did I understand why the vendir githubResource was not able to download it. If my project chooses to use vendir to track upstream OSS dependencies via githubResources, we probably want to track them uniformly, even if they don't ship any specific assets.

I would expect to be able to choose the tarball_url or zipball_url as options to download, possibly by specifying their paths as they appear when you download them via the github UI: https://github.com/open-policy-agent/gatekeeper/archive/v3.3.0.zip and https://github.com/open-policy-agent/gatekeeper/archive/v3.3.0.tar.gz

Paths within the sourcecode for the release would also be nice, since in that case we'd probably only need metadata from a small number of files within the release.

Environment:

• vendir version (execute vendir --version): 0.16.0
• OS (e.g. from /etc/os-release): MacOS

[cppforlife]

@LukeWinikates any thoughts on picking git (by a tag) vs githubRelease fetch types to fetch source code?

[LukeWinikates]

I think using the git fetch type is a viable workaround. Honestly, for our use case, there doesn't seem to be anything in the source code that we actually need from the git repo either, but we were attempting to conform to a vendir usage pattern established by the codebase we were contributing to.

Other packages in the repository we were adding to had a pattern of using vendir to pull in yaml files for Kubernetes deployments, which then fed into using imgpkg to resolve the image versions in those yamls into a specific image sha. We wanted to follow that same pattern to document our upstream open-source dependency, but it turns out that none of our dependencies ship yaml deployment manifests with their releases, so the vendir -> yaml -> imgpkg -> resolved sha pipeline didn't pan out in those cases.

We ended up not using vendir at all for those packages - the downside is that for other packages in that repository the vendir.yaml file is the source of truth for their upstream dependencies, whereas for our contributions, it's going to be maintained by editing the individual image tags in the yaml files we author ourselves.

I will say that the error message in this case made me think I was getting something wrong in my yaml paths, so I spent a fair amount of time thinking the problem was something on my local filesystem. I took a fair amount of time and trial and error to figure out that it was actually the json response of the specific release that was causing the error. Detecting the specific case of a release with no assets and printing an error message that says "error: no files in release, try a git resource instead`" would be a good thing to do, I think. Happy to submit a PR for that, if it would be welcomed.

[cppforlife]

"error: no files in release, try a git resource instead`" would be a good thing to do, I think. Happy to submit a PR for that, if it would be welcomed.

i think that would be a helpful addition.

it's going to be maintained by editing the individual image tags in the yaml files we author ourselves.

so it sounds like the asset that you actually care about is the image tag and that image tag is part of somebody else yaml file. hmm ill have to chew on that if vendir can be (/should be) helpful in such scenario (i could imagine having vendir provide some metadata that could be used to "generate" a file that would be a ytt overlay -- sounds a little weird... will think more)?

[github-actions[bot]]

This issue is being marked as stale due to a long period of inactivity and will be closed in 5 days if there is no response.

[github-actions[bot]]

This issue is being marked as stale due to a long period of inactivity and will be closed in 5 days if there is no response.