CRDs upgrade support

[shahar-h]

Description: Currently Gateway API and EG CRDs are located in EG gateway-helm chart crds folder. This means that CRDs are not upgraded on chart upgrade, and need to be manually upgraded beforehand.

Possible solutions:

1. Move crds inside templates. Real world example: cert-manager chart has CRDs as part of templates folder and exposes crds.enabled flag. It also allows to decide if we want to keep the CRDs on chart uninstallation with crds.keep flag, by leveraging "helm.sh/resource-policy": keep helm annotation. See: https://cert-manager.io/v1.13-docs/installation/helm/#crd-considerations. However, this is a breaking change for consumers that use EG chart as a sub chart and have custom resources as part of the main chart.
2. Split the current chart into crds chart and application chart, and instruct to install the crds chart first. Real world example: Istio provides a base chart that installs CRDs: https://istio.io/latest/docs/setup/install/helm/#installation-steps.

Both 1 and 2 can also solve https://github.com/envoyproxy/gateway/issues/3094 by providing a separate flags(1)/charts(2) for Gateway API and EG CRDs.

Also Related:

• https://github.com/envoyproxy/gateway/issues/2031
• https://github.com/envoyproxy/gateway/issues/1721

WDYT?