Pulling a nested bundle should allow filtering

[DennisDenuto]

Use Case

As an Application consumer When consuming a Bundle that contains lots of Nested Bundles I want control over which Nested Bundles are pulled So that less compute and storage resources are consumed And to reduce the time it takes to pull Nested Bundles

Description

Introduce a new flag --recursive-filter to the pull command, accepting multiple filters to apply when pulling a Bundle.

A Filter is a key-value that matches against an Images annotation contained in a Bundle's ImageLock file. If the image's annotation matches the provided filter/s then the image (which will always be a bundle) is fetched. Otherwise it is skipped.

Acceptance Criteria

Given the following Bundle with Nested Bundle relationship:

my.registry.io/bundle-1 -> my.registry.io/bundle-2 -> my.registry.io/bundle-3
                        -> my.registry.io/ubuntu

And

The following bundles are structured:

• my.registry.io/bundle-1@sha256:123 The .imgpkg/images.yml looks like the following

  apiVersion: imgpkg.carvel.dev/v1alpha1
  kind: ImagesLock
  images:
  - image: my.registry.io/bundle-2@sha256:456
    annotations:
      BUNDLE-2-ANNOTATION: "true"
  - image: my.registry.io/ubuntu@sha256:999
    annotations:
      IMAGE-UBUNTU-ANNOTATION: "true"
  

• my.registry.io/bundle-2@sha256:456 The .imgpkg/images.yml looks like the following

  apiVersion: imgpkg.carvel.dev/v1alpha1
  kind: ImagesLock
  images:
  - image: my.registry.io/bundle-3@sha256:789
     annotations:
       BUNDLE-3-ANNOTATION: "true"
  

The following scenarios should be fulfilled: 🟢 Scenario 0: Every image referenced by the 'root' Bundle is fetched

imgpkg pull -b my.registry.io/bundle-1 --recursive

Results in every bundle being fetched.

Expected Standard Output:

Pulling bundle 'my.registry.io/bundle-1'
  Extracting layer 'sha256:111' (1/1)

Pulling Nested Bundle 'my.registry.io/bundle-2@sha256:456'
  Extracting layer 'sha256:111' (1/1)

 Pulling Nested Bundle 'my.registry.io/bundle-3@sha256:789'
   Extracting layer 'sha256:111' (1/1)

🟢 Scenario 1: Only first level Nested Bundle is fetched

imgpkg pull -b my.registry.io/bundle-1 --recursive --recursive-filter BUNDLE-2-ANNOTATION=true

Results in only bundle-2 image being fetched.

Expected standard output

Pulling bundle 'my.registry.io/bundle-1'
  Extracting layer 'sha256:111' (1/1)

Pulling Nested Bundle 'my.registry.io/bundle-2@sha256:456'
  Extracting layer 'sha256:111' (1/1)

  Skipped Pulling Nested Bundle 'my.registry.io/bundle-3@sha256:789' (due to recursive-filter flag)

🟢 Scenario 2: Every Nested Bundle is fetched

imgpkg pull -b my.registry.io/bundle-1 --recursive --recursive-filter BUNDLE-2-ANNOTATION=true --recursive-filter BUNDLE-3-ANNOTATION=true

Results in every bundle being fetched.

Expected Standard Output:

Pulling bundle 'my.registry.io/bundle-1'
  Extracting layer 'sha256:111' (1/1)

Pulling Nested Bundle 'my.registry.io/bundle-2@sha256:456'
  Extracting layer 'sha256:111' (1/1)

 Pulling Nested Bundle 'my.registry.io/bundle-3@sha256:789'
   Extracting layer 'sha256:111' (1/1)

🟢 Scenario 3: Every Nested Bundle is skipped

imgpkg pull -b my.registry.io/bundle-1 --recursive --recursive-filter BUNDLE-3-ANNOTATION=true

Results in every nested bundle skipped. This is because the intermediate Bundle (Bundle-2) was not fetched. And thus, the ImagesLock file contained in Bundle-2 was not read / processed.

Pulling bundle 'my.registry.io/bundle-1'
  Extracting layer 'sha256:111' (1/1)

Skipped Pulling Nested Bundle 'my.registry.io/bundle-2@sha256:456' (due to recursive-filter flag)

🟢 Scenario 4: Filter flag is used, but the recursive flag is not provided

imgpkg pull -b my.registry.io/bundle-1 --recursive-filter BUNDLE-2-ANNOTATION=true

Results in only the root bundle 'bundle-1' being fetched.

Expected standard output

Pulling bundle 'my.registry.io/bundle-1'
  Extracting layer 'sha256:111' (1/1)

🔴 Scenario 5: Filter flag is incorrectly used

imgpkg pull -b my.registry.io/bundle-1 --recursive --recursive-filter

or

imgpkg pull -b my.registry.io/bundle-1 --recursive --recursive-filter ONLY_KEY_PROVIDED

Results in imgpkg pull command error-ing out.

Example of a possible standard Error message given to the user

imgpkg pull error: --recursive-filter applies against keys and values. Only the key was provided.

Notes:

• Filtering on key-values are case sensitive.

Out of scope:

• wildcard value matching
• defaulting an empty value in the filter e.g. --recursive-filter annotation_key=
• UX warning messages around using --recursive-filter and not providing --recursive flag. See: https://github.com/vmware-tanzu/carvel-imgpkg/issues/118

[cppforlife]

note that annotations type is map[string]string, so BUNDLE-2-ANNOTATION: true would not be valid value.

--filter-bundle-by-annotation

i would propose something like -r --recursive-filter BUNDLE-2-ANNOTATION=.