Package Generation for projects

[brandtkeller]

Is your feature request related to a problem? Please describe. In order to support OSS projects (and their maintainers) and the inclusion of Air Gap testing and validation of that project using Zarf, it would be valuable if Zarf could generate the zarf.yaml file based on a provided git repository containing a helm chart.

(This has other possible value - such as it relates to #513 - but I wanted to provide a specific scenario and persona to provide for initial logic.)

This logic could be a repeatable way to abstract zarf knowledge required by OSS maintainers into automation that can generate a file that can easily be consumed by end users with zarf package create.

Ideally this could be valuable during release candidate testing - but also flow into testing of air gap in CI.

Describe the solution you'd like Execute zarf package generate <helm chart repo git url/helm registry url> (or similar) to begin the generation of a zarf.yaml file. This command would use logic such as that in zarf prepare find-images to find the images in the chart, and otherwise use the helm chart target to instantiate the chart required fields.

Describe alternatives you've considered None - This feature is proposed to help assist OSS projects understand air gap considerations and test/validate them without needing to be a zarf expert.

Additional context Discussions around supporting open source and the ability to test air gap in CI spurred thiis thought.

[corang]

I'm gonna create a branch and start experimenting with things, I'm thinking something along the lines of zarf package generate <package_name> <flags/components> with different flags for each kind of component.

[corang]

After doing some thinking I'm leaning away from flags and more of an interactive wizard to help create the package and explain what/why its asking/doing things

[jeff-mccoy]

We actually will need both as that's how the zarf cli works. Wizards are fine, but always have a way to run full automation via flag/config file.

[corang]

I'm not sure that possible in a reasonable way, if you want to include a remote chart from a repo in this generated zarf.yaml how do you specify that in a not horrible way, you need to have the repo url, the chart name, and the chart version

[corang]

Thoughts on this for the command line? @jeff-mccoy @brandtkeller To be honest I'm not really happy with it but its difficult to find a good way to get this kind of data in through the command line

Usage:
zarf package generate [package_name] [flags]

Flags:

-n, --component-name       'The name of the component (string)'
-r, --required             'Is this component required (true,false)'
-t, --component-data-type  'The kind of component data (chart, manifest, other)'
-d, --component-data       'JSON data for given component data type, keys/data match zarf.yaml schema. Data conforms to 
given --component-data-type ([{"name": "chart-name", "namespace": "chart-namespace", "version": "0.1", 
"localPath": "/path/to/chart" }])'

Flag Usage:

'Each instance of "--component-name" creates a new component. Every component needs at least one "--component-type". If 
the component type is not "other" then "--component-data" is required. If component type is "other" then "--component-data" 
is not allowed. Component data is an array of objects in json, these objects must conform to the schema for a given 
component data type. A component can have multiple component data types, but not any duplicates.'

Examples:

zarf package generate charts_only --component-name "some_charts" --component-data-type "chart" --component-data 
'[{"name": "chart1", "namespace": "ns1", "version": "1.0", "url": "https://chart.repo.site/reponame"},{"name": "chart2", 
"namespace": "ns2", "version": "1.0", "url": "https://chart.repo.site/reponame"}]'

zarf package generate manifests_only --component-name "some_manifests" --component-data-type "manifest" 
--component-data '[{"name": "manifests1", "namespace": "ns1", "files": ["manifest1.yaml","manifest2.yaml"]},{"name": 
"manifests2", "namespace": "ns2", "files": ["manifest3.yaml","manifest4.yaml"]}]'

zarf package generate charts_manifests --component-name "manifests_and_charts" --component-data-type "manifest" 
--component-data '[{"name": "manifests1", "namespace": "ns1", "files": ["manifest1.yaml","manifest2.yaml"]},{"name": 
"manifests2", "namespace": "ns2", "files": ["manifest3.yaml","manifest4.yaml"]}]' --component-data-type "chart" 
--component-data '[{"name": "chart1", "namespace": "ns1", "version": "1.0", "url": "https://chart.repo.site/reponame"},
{"name": "chart2", "namespace": "ns2", "version": "1.0", "url": "https://chart.repo.site/reponame"}]'

zarf package generate multi_component --component-name "chart_component_1" --component-data-type "chart" 
--component-data '[{"name": "chart1", "namespace": "ns1", "version": "1.0", "url": "https://chart.repo.site/reponame"}]' 
--component-name "chart_component_2" --component-data-type "chart" --component-data '[{"name": "chart2_1", 
"namespace": "ns2_1", "version": "1.0", "url": "https://chart.repo.site/otherrepo"}]'

zarf package generate other_package --component-name "empty component" --component-data-type "other"

zarf package generate empty_package

[brandtkeller]

Notes from discussion:

The functionality of a wizard may be very valuable to the primary persona of Zarf (could use validation here).

We want to provide the CLI-only execution in order to be compliant with zarf automation. This also provides a base for which to test the functionality (as there may not be an easy method for testing wizard input at this time).

Likely we would want to target some other execution that does not use package - as package is terminology around the archive created and not the metadata that feeds the creation of the package.

[corang]

~~Looks like viper doesn't support my whole idea of inter-dependent and order important flags as specified above, trying to think of an alternative that isn't just all json~~

Scratch that, forgot I can just use the go builtin functions to get what I need

[lucasrod16]

Using flags for users to populate the zarf.yaml feels a bit counterintuitive. It assumes the user has a good understanding of how zarf.yaml files are structured, while also requiring the user to learn some non-trivial command line flags.

I think an important aspect of this work is summed up nicely in the issue description by @brandtkeller:

This feature is proposed to help assist OSS projects understand air gap considerations and test/validate them without needing to be a zarf expert.

The initial feature request for the generation of a zarf package had this as a primary goal as well.

The simplicity and utility of the helm create command to generate a helm chart that is ready to package, install, and otherwise explore/learn about helm and helm charts is the same desired outcomes we want for users with zarf. A simple core command (with maybe a single flag if necessary) that gives you a zarf.yaml that is ready to use to create, deploy, and otherwise explore/learn about zarf and zarf packages. All without needing any existing knowledge of what a zarf.yaml file looks like.

With the goal of generating a zarf.yaml file from a helm chart in mind, I think a discussion around how we can go about doing that would be helpful. This issue also mentions using logic from zarf prepare find-images, and there have been discussions about how that logic can be flaky. In particular, with charts that deploy operators, the container images are often embedded in the operator binary, thus never being discovered. This is an obvious problem in this case, because all artifacts must be included and accounted for in the zarf.yaml to ensure you have all the bits you need when you create your zarf package.

I'm not sure how common it is for helm charts to deploy operators with embedded images. If it is a small number compared to those that do not use this pattern, then we may be able to consider it an edge case that simply hasn't been solved, while still providing value for the majority that don't do it.

Love the idea and intention behind package generation, so looking forward to seeing how this continues to evolve as we iterate.

[corang]

While I did create a CLI pretty much according to my comment above I do agree that it's quite counter-intuitive and not really useful. I'm going to leave that code on the branch for now but move towards something similar to the angular ng generate command (https://angular.io/cli/generate). It will be somewhat similar in that it will modify/create different parts of a zarf.yaml depending on which command you pass it, a full walkthrough wizard I still think has value and will be part of it. Current concept for this:

Zarf package generation wizard and CLI.

Usage:
  zarf generate [COMMAND] [flags]

Available Commands:

  wizard      Start the package creation wizard
  package     Create a new empty package (zarf.yaml)
  component   Create or modify a component
  image       Create or modify an image entry
  constant    Create or modify a constant
  variable    Create or modify a variable

Flags:

  -f, --path string   Path to zarf.yaml to modify/create (default "./zarf.yaml")

Examples:

  zarf generate wizard

  zarf generate package package_name

  zarf generate component component_name

  zarf generate component component_name.charts.chart_name --url url --gitPath path

[Racer159]

@corang / @lucasrod16 / @andrewg-xyz is this a dupe of #513 ?

[lucasrod16]

@corang / @lucasrod16 / @andrewg-xyz is this a dupe of #513 ?

@Racer159 Yes, I think this issue was an attempt to re-frame the original issue I opened. The intent and desired outcomes are identical though

[Racer159]

Adding some Gherkin to this to encapsulate what this first stab should do:

As Ezra I want a way to generate a zarf.yaml from a Helm chart (starting with charts in git) so that I can create boilerplate for my package in an easier way.

Given I have a chart in a git repository When I run zarf dev generate <package-name> --url <git-url> --version <version> --gitPath <sub-folder> Then Zarf will create a definition with that package name, chart and associated images

As an example:

zarf dev generate podinfo --url https://github.com/stefanprodan/podinfo.git --version 6.4.0 --gitPath charts/podinfo

Would make the following:

kind: ZarfPackageConfig
metadata:
  name: podinfo # This is <package-name>

components:
  - name: podinfo # This is <package-name>
    required: true
    charts:
      - name: podinfo # This is <chart-name> from the Chart.yaml
        version: 6.4.0 # This is --version
        namespace: podinfo # This is <chart-name> from the Chart.yaml
        url: https://github.com/stefanprodan/podinfo.git # This is --url
        gitPath: charts/podinfo # This is --gitPath
    images: # These are autodiscovered with find-images logic
      - ghcr.io/stefanprodan/podinfo:6.4.0
      # This is the cosign signature for the podinfo image for image signature verification
      - ghcr.io/stefanprodan/podinfo:sha256-57a654ace69ec02ba8973093b6a786faa15640575fbf0dbb603db55aca2ccec8.sig

[Racer159]

Also this should be marked with [beta] in the command description / docs