feat: Vendir manager

[gberche-orange]

What would you like Renovate to be able to do?

Vendir is a tool to vendor a fair variety of resources on the local file system that map to renovate datasources:

https://carvel.dev/vendir/docs/v0.30.0/

Supported sources for fetching: git image (image from OCI registry) githubRelease helmChart

Sample:

apiVersion: vendir.k14s.io/v1alpha1
kind: Config
directories:

- path: k8s-manifests 
  contents:

  - path: github.com/vmware-tanzu/carvel-kapp-controller
    githubRelease:
      slug: vmware-tanzu/carvel-kapp-controller
      tag: v0.30.0

Full syntax spec: https://carvel.dev/vendir/docs/v0.30.0/vendir-spec/

Currently, I'm using the regexp manager with the following rules, however a built-in manager would enable more efficient sharing with the community

  "regexManagers": [
    {
      "fileMatch": ["vendor/vendir-k8s-manifest.yml$"],
      "matchStrings": [
        " +githubRelease:\\n +slug: (?<depName>[a-z0-9\\.\\/\\-]*?)\\n +tag: (?<currentValue>[a-z0-9\\.\\/\\-\\+]*?) *\\n/"
      ],
      "datasourceTemplate": "github-releases",
      "versioningTemplate": "regex:v?(?<major>\\d+)\\.(?<minor>\\d+)\\.(?<patch>\\d+)?$"
    },
    {
      "description": "https://regex101.com/r/SChCZX/1",
      "fileMatch": ["vendor/vendir-k8s-manifest.yml$"],
      "matchStrings": [
        " +git:\\n +url: (?<depName>[a-z0-9\\.\\/\\-:]*?)\\n +ref: (?<currentValue>[a-z0-9\\.\\/\\-\\+]*?) *\\n"
      ],
      "datasourceTemplate": "git-refs",
      "versioningTemplate": "regex:v?(?<major>\\d+)\\.(?<minor>\\d+)\\.(?<patch>\\d+)?$"
    }
  ]

If you have any ideas on how this should be implemented, please tell us here.

a new manager

Is this a feature you are interested in implementing yourself?

Maybe

[github-actions[bot]]

Hi there,

You're asking us to support a new package manager. We need to know some basic information about this package manager first. Please copy/paste the new package manager questionnaire, and fill it out in full.

Once the questionnaire is filled out we'll decide if we want to support this new manager.

Good luck,

The Renovate team

[gberche-orange]

@rarkins here is the questionnaire filled.

New package manager questionnaire

Did you read our documentation on adding a package manager?

• [x] I've read the adding a package manager documentation.

Basics

Name of package manager

Vendir

What language does this support?

It supports datasource rather than programming languages

git image (image from OCI registry) githubRelease helmChart

How popular is this package manager?

1K code match on github for the manifest header https://github.com/search?p=2&q=vendir.k14s.io%2Fv1alpha1&type=Code

Does this language have other (competing?) package managers?

• [ ] Yes (give names)
• [ ] No
• [x] Not sure

---

Package File Detection

What type of package files and names does it use?

vendir.yml

What fileMatch pattern(s) should be used?

/vendir\.yml$

Is it likely that many users would need to extend this pattern for custom file names?

• [ ] Yes
• [x] No

Is the fileMatch pattern likely to get many "false hits" for files that have nothing to do with package management?

No

---

Parsing and Extraction

Can package files have "local" links to each other that need to be resolved?

No

Is there a reason why package files need to be parsed together (in serial) instead of independently?

No

What format/syntax is the package file in?

• [ ] JSON
• [ ] TOML
• [x] YAML
• [ ] Custom (explain below)

How do you suggest parsing the file?

• [ ] Off the shelf parser
• [ ] Using regex
• [ ] Custom-parsed line by line
• [x] Other: yaml parser + simple filtering such as yq, or native nodejs

Does the package file structure distinguish between different "types" of dependencies? e.g. production dependencies, development dependencies, etc?

• [ ] Yes, production and development dependencies
• [x] No, all dependencies are treated the same

List all the sources/syntaxes of dependencies that can be extracted

• git: git-refs
• image (image from OCI registry): docker
• githubRelease: github-releases
• helmChart: helm

Describe which types of dependencies above are supported and which will be implemented in future

All are supported

Versioning

What versioning scheme does the package file(s) use?

• semver

Does this versioning scheme support range constraints, e.g. ^1.0.0 or 1.x?

• [x] Supports range constraints (e.g ^1.0.0 or 1.x)
• [ ] No

Is this package manager used for applications, libraries, or both? If both, is there a way to tell which is which?

• [ ] Applications
• [ ] Libraries
• [x] Both (explain how to tell which is which)

It is used to vendor arbitrary files locally, typically in a git repo. This can then be used for vendoring libraries, or vendoring applications (such as helm charts)

If ranges are supported, are there any cases when Renovate should pin ranges to exact versions if rangeStrategy=auto?

I'm not sure, likely no

Lookup

Is a new datasource required? Provide details

• [ ] Yes, provide details.
• [x] No.

Will users need the capability to specify a custom host/registry to look up? Can it be found within the package files, or within other files inside the repository, or would it require Renovate configuration?

registries are specified within the package file (vendir.yaml)

Do the package files have any "constraints" on the parent language (e.g. supports only v3.x of Python) or platform (Linux, Windows, etc.) that should be used in the lookup procedure?

No

Will users need the ability to configure language or other constraints using Renovate config?

No

---

Artifacts

Are lock files or checksum files used? Are they mandatory?

vendor.lock.yaml file are used by vendir, see https://carvel.dev/vendir/docs/v0.30.0/vendir-lock-spec/

If so, what tool and exact commands should be used if updating one or more package versions in a dependency file?

Just replace the currentVersion in the file.

If applicable, describe how the tool maintains a cache and if it can be controlled via CLI or environment variables? Do you recommend the cache be kept or disabled/ignored?

vendir sync command would update the vendored files along with the vendor.lock.yaml file

If applicable, what command should be used to generate a lock file from scratch if you already have a package file? This will be used for "lock file maintenance"

vendir sync command

Other

Is there anything else to know about this package manager?

See https://carvel.dev/vendir/docs/v0.30.0/

[dominik-horb-umg]

@gberche-orange Does your regexManager update only the vendir configuration file or did you also find a way to actually run vendir already?

I'd also be interested in this feature and could imagine helping with the implementation.

[gberche-orange]

@dominik-horb-umg No, I did not manage to run vendir sync , the regexmanager rules only update the vendir.yaml file

[dominik-bln]

I've started to do a bit of work to add the new manager in the above draft PR. It's very basic for now and only extracts vendir dependencies of type helmChart as a first step, so that the initial PR can stay small. Will try to clean it up and add docs in the coming days.

[viceice]

@dominik-bln for lockfile updates we need a new installer for vendir at containerbaee/base repo first.

[dominik-bln]

@viceice Thanks. I've opened https://github.com/containerbase/base/pull/657 to address that.

[dominik-bln]

Current status is that the vendir tool install in the containerbase/base repo is merged and extraction from vendir.yml is working in the draft PR. Next I'll try to get the lock file maintenance working.

Here is a basic test repo if anyone else wants to take a look:

https://github.com/dominik-bln/renovate-vendir-test

[knechtionscoding]

@dominik-bln figured I'd check in and see if you had made any more progress/need help moving the basic implementation working?

[dominik-horb-umg]

Sadly no progress from my side. IIRC the basic implementation without lockfile maintenance is working, so we could probably get that into a mergeable state and iterate from there if it helps. That would just replace something that can be done with a Regex Manager though.

[knechtionscoding]

@dominik-bln I know you have the PR out, if you have time and can get it to a mergable state then I can iterate and help with the lockfile part.

Alternatively if you don't have time then I can submit a new PR building on your branch and get it working.

[knechtionscoding]

@dominik-bln I also think that for vendir we could follow a similar model to package.json:

If you use a lock file then changes to your package file must come with a compatible change to the lock file. Renovate can patch/update package files directly, but can't "reverse engineer" lock files. This is why Renovate lets the package manager do the lock file update. A simplified example:
- The repository has a package.json and package-lock.json with version 1.0.0 of a dependency
- Renovate sees that version 1.1.0 is available
- Renovate patches the package.json to change the dependency's version from 1.0.0 to 1.1.0
- Renovate runs npm install to let npm update the package-lock.json
- Renovate commits the package.json and package-lock.json
- Renovate creates the PR

In the case of vendir it would be:

• update vendir.yml
• run vendir sync
• commit either all changed files or just vendir.yml and vendir.lock.yml

[renovate-release]

:tada: This issue has been resolved in version 37.264.0 :tada:

The release is available on:

• GitHub release
• npm package (@latest dist-tag)
• 37.264.0

Your semantic-release bot :package::rocket: