feat: Expanding generator resources

[davis-haba]

What this PR does / why we need it:

This PR implements the "gator expand" CLI command, and adds expansion functionality to audit and the validation webhook.

Design Doc: https://docs.google.com/document/d/1wsTnP0IYNcwVxy7XVs8WRv4WUhJZPJl_RdVZaGd5LNc/edit

Which issue(s) this PR fixes (optional, using fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when the PR gets merged): Fixes #

Special notes for your reviewer:

[codecov-commenter]

Codecov Report

Base: 54.63% // Head: 53.38% // Decreases project coverage by -1.25% :warning:

Coverage data is based on head (418c0bb) compared to base (f21d07a). Patch coverage: 35.34% of modified lines in pull request are covered.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2062      +/-   ##
==========================================
- Coverage   54.63%   53.38%   -1.26%     
==========================================
  Files         111      115       +4     
  Lines        9556    10131     +575     
==========================================
+ Hits         5221     5408     +187     
- Misses       3938     4307     +369     
- Partials      397      416      +19     

Flag	Coverage Δ
unittests	53.38% <35.34%> (-1.26%)	:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
pkg/audit/controller.go	0.00% <0.00%> (ø)
pkg/audit/manager.go	0.00% <0.00%> (ø)
pkg/controller/config/config_controller.go	63.82% <0.00%> (-0.28%)	:arrow_down:
pkg/controller/constraint/constraint_controller.go	5.66% <0.00%> (-0.02%)	:arrow_down:
...onstrainttemplate/constrainttemplate_controller.go	58.85% <0.00%> (-0.15%)	:arrow_down:
...controller/externaldata/externaldata_controller.go	56.45% <0.00%> (-0.93%)	:arrow_down:
pkg/gator/expand.go	0.00% <0.00%> (ø)
pkg/gator/filereader.go	0.00% <0.00%> (ø)
pkg/gator/read_constraints.go	84.16% <0.00%> (-2.16%)	:arrow_down:
...mutation/mutators/assignmeta/assignmeta_mutator.go	29.80% <0.00%> (-1.51%)	:arrow_down:
... and 21 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

[sozercan]

can we add e2e tests and documentation?

[sozercan]

seeing this in the logs (there's no ExpansionTemplate.mutations.gatekeeper.sh)

{"level":"error","ts":1660088328.3324754,"logger":"controller-runtime.source","msg":"if kind is a CRD, it should be installed before calling Start","kind":"ExpansionTemplate.mutations.gatekeeper.sh","error":"no matches for kind \"ExpansionTemplate\" in version \"mutations.gatekeeper.sh/v1alpha1\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/source.(*Kind).Start.func1.1\n\t/go/src/github.com/open-policy-agent/gatekeeper/vendor/sigs.k8s.io/controller-runtime/pkg/source/source.go:139\nk8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext\n\t/go/src/github.com/open-policy-agent/gatekeeper/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:233\nk8s.io/apimachinery/pkg/util/wait.WaitForWithContext\n\t/go/src/github.com/open-policy-agent/gatekeeper/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:660\nk8s.io/apimachinery/pkg/util/wait.poll\n\t/go/src/github.com/open-policy-agent/gatekeeper/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:594\nk8s.io/apimachinery/pkg/util/wait.PollImmediateUntilWithContext\n\t/go/src/github.com/open-policy-agent/gatekeeper/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:545\nsigs.k8s.io/controller-runtime/pkg/source.(*Kind).Start.func1\n\t/go/src/github.com/open-policy-agent/gatekeeper/vendor/sigs.k8s.io/controller-runtime/pkg/source/source.go:132"}

[sozercan]

@davis-haba I think we talked about default warn enforcement action for MVP before. what are we defaulting here?

[ritazh]

Should we feature gate this like all other new features?

[maxsmythe]

re: "ExpansionTemplate.mutations.gatekeeper.sh" that was the grouping the resource was originally stuck in. It looks like there is a place where a watch for that key isn't cleaned up.

@davis-haba can you do a search through this PR and make sure this (and any other dead code) is removed?

@sozercan re: default enforcement action. I don't think we are currently setting one. Given the current size of the PR and that this feature wont be enabled by-default (either because of a feature gate or b/c we aren't shipping expansion resources), are you okay with the ability to override EA for expanded resources being a follow-on PR?

@ritazh What level of flag-gating were you thinking? By-default this feature doesn't do anything unless there are resources. Do we want to flag-gate initializing the controller so that: (1) users can avoid pushing the ExpansionTemplate CRD without errors and (2) removing the controller as a vector for adding expansions and activating the feature? Does that SGTY?

[ritazh]

Do we want to flag-gate initializing the controller so that: (1) users can avoid pushing the ExpansionTemplate CRD without errors and (2) removing the controller as a vector for adding expansions and activating the feature? Does that SGTY?

SGTM

[maxsmythe]

It looks like DCO is failing due to the inline requests from @ritazh . @ritazh do you know why GitHub doesn't add the signoff message? It says that it will.

[davis-haba]

Documentation added in a separate PR:

https://github.com/open-policy-agent/gatekeeper/pull/2229

[maxsmythe]

breaking tests due to disabled-by-default... will fix

[maxsmythe]

Not sure why the DCO became re-unhappy, rebasing on davis's previous commit

[maxsmythe]

Oh, DCO was set to manually approved... that shouldn't have been done

[maxsmythe]

Needed to sign/rebase every commit individually to avoid resigning all commits and re-resolving merge conflicts

[maxsmythe]

For some reason a non-signed off commit bumping golang is also in this branch (signed off equivalent in main) I have no idea how this got to this state: c1d0d94

[maxsmythe]

For some reason, commits from main were interspersed with commits to the release branch and merge commits. This made it impossible to rebase or otherwise clean up the non-signed-off commit.

Created a new branch off of master and cherrypicked all PR-related commits onto that branch. There was a small diff in the end due to my decisions on resolving merge conflicts, so added a commit to remove that diff.

Finally, switched the original branch to point to the new branch.

The result is a clean base with only PR-related commits on top of our main branch.

[maxsmythe]

@davis-haba all green now, stopping working on your branch. Thank you for letting me mangle it.

@ritazh feature flag added

@sozercan your most recent spate of comments (re: the added tests) have not been addressed.

[maxsmythe]

@sozercan @ritazh all comments should be addressed.

[sozercan]

@davis-haba looks like there's a merge conflict now

[sozercan]

waiting for @ritazh's review

[ritazh]

/hold confirming latest audit changes