[bitnami/external-dns] External-DNS Chart Broken with 7.3.3 due to removal of protected CRD group annotation

[pinkfloydx33]

Name and Version

bitnami/external-dns

What architecture are you using?

None

What steps will reproduce the bug?

Attempt to upgrade or install the external-dns chart at version 7.3.3. In our case, an automated upgrade via Flux has started spamming alerts across all of our environments where minor/patch upgrades are automatically performed.

What is the expected behavior?

Helm chart upgrades/installs

What do you see instead?

Installation fails with:

Helm upgrade failed: cannot patch "dnsendpoints.externaldns.k8s.io" with kind CustomResourceDefinition: CustomResourceDefinition.apiextensions.k8s.io "dnsendpoints.externaldns.k8s.io" is invalid: metadata.annotations[api-approved.kubernetes.io]: Required value: protected groups must have approval annotation "api-approved.kubernetes.io", see https://github.com/kubernetes/enhancements/pull/1111

Additional information

This is because the automated upgrade has removed the protected annotation on the CRDs:

• Commit @ Line12
• PR

This annotation is required on CRDs if the group is k8s.io, kubernetes.io, or ends with .k8s.io, .kubernetes.io which is applicable here because the CRD group is apiextensions.k8s.io.

This change needs to be reverted or else the chart is unusable.

[carrodher]

Thank you for bringing this issue to our attention. We appreciate your involvement! If you're interested in contributing a solution, we welcome you to create a pull request. The Bitnami team is excited to review your submission and offer feedback. You can find the contributing guidelines here.

Your contribution will greatly benefit the community. Feel free to reach out if you have any questions or need assistance.

[pinkfloydx33]

@carrodher I wouldn't know where to begin as the original change was made by some form of automation, which tells me that it too would need to be fixed. I also unfortunately need approval from my company before contributing, even to tools we use, and that would likely take more time than others broken by this change would appreciate.

[andeke07]

I have added the annotation to the pull request on the original external-dns project, I believe this is where the Bitnami automation gets its information from so if this gets approved hopefully the next release will contain it again?

[andeke07]

This has been fixed now with https://github.com/kubernetes-sigs/external-dns/commit/f46676fdbbe336cc3c86f68d5c96c7cf920aeba0

I don't know the next steps though, I suppose a release needs to be done and then the bitnami chart updated to refer to the new release?

[rouke-broersma]

I think the release of such updates is automated, not sure how they are triggered but I expect that it will come within a reasonable time.

[carrodher]

Our automation looks for new releases at https://github.com/kubernetes-incubator/external-dns. When a new release is cut there, our automated test & release process is triggered. As part of that process, the upstream CRDs are compared and the ones in the Bitnami chart are updated to match the upstream, i.e https://github.com/bitnami/charts/pull/25962/commits/375ee3ba1adfabacb8c8ccb29fb97b78c15e92c1

[herrbpl]

Still broken in 7.5.0

[andeke07]

external-dns hasn't issued a new release yet so there's nothing for the new chart to go off of

[raiomarco]

So... there's a fix? a workaround? if not, what version of the chart should i use? i tried the 7.5.2 but it didn't work :(

[cheinema]

@raiomarco Since the problem seems to have occurred starting with chart version 7.3.3, v7.3.2 should be the last stable version for now. We are still waiting for a new release in https://github.com/kubernetes-sigs/external-dns to include the fix.

[MaxAnderson95]

@cheinema Unless I'm missing something, the v0.14.2 release appears to have the fix. Is there anything else that needs to be done before merging the fix into the chart?

[rouke-broersma]

@cheinema Unless I'm missing something, the v0.14.2 release appears to have the fix. Is there anything else that needs to be done before merging the fix into the chart?

You're looking at the external dns helm chart managed by external dns. This is not the source of crds for the bitnami chart.

[MaxAnderson95]

@rouke-broersma I knew I was missing something! Thanks.

[Atoms]

so seems crd cannot be updated manually, as there is ci pipeline which allows only bitnami bot to update crd. and there is no release from external-dns side which would include api-approved annotation.

[pinkfloydx33]

This chart is effectively broken for now... Does anyone have a way to workaround it or are we SOL for now?

[rouke-broersma]

This chart is effectively broken for now... Does anyone have a way to workaround it or are we SOL for now?

I think you could simply deploy the required resources yourself (updated crd, clusterRole on crd, clusterRoleBinding to serviceaccount from chart): https://github.com/search?q=repo%3Abitnami%2Fcharts%20path%3A%2F%5Ebitnami%5C%2Fexternal-dns%5C%2Ftemplates%5C%2F%2F%20.Values.crd.create&type=code

That should be sufficient until upstream releases a new version.

[pinkfloydx33]

Ok thanks. We use Flux for management, I'm sure theres a way to do that, just haven't looked into it yet. Hopefully the upstream fixes it soon...

[hawkesn]

Still broken with 7.5.7

[andeke07]

@hawkesn the chart relies on external-dns coming out with another release which they haven't done in a month: https://github.com/kubernetes-sigs/external-dns/releases

The next time external-dns release, it will have the fix, which will then get embedded in this chart.

[github-actions[bot]]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[MaxAnderson95]

Commenting to keep this issue open

[github-actions[bot]]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[MaxAnderson95]

Commenting to keep this issue open ...again

[venkatamutyala]

Commenting to keep this active and opened an issue here requesting a release be cut: https://github.com/kubernetes-sigs/external-dns/issues/4657

[mpechner-akasa]

This is real.

[Mykyta-Serbynevskyi]

Hi there, This issue also impacts our services, as of now it is not possible to upgrade the external-dns helmchart from 7.2.1 to latest 8.3.4:

>  helm upgrade -i external-dns bitnami/external-dns -n clusterutils -f values-external-dns.yaml --version 8.3.4
Error: UPGRADE FAILED: cannot patch "dnsendpoints.externaldns.k8s.io" with kind CustomResourceDefinition: CustomResourceDefinition.apiextensions.k8s.io "dnsendpoints.externaldns.k8s.io" is invalid: metadata.annotations[api-approved.kubernetes.io]: Required value: protected groups must have approval annotation "api-approved.kubernetes.io", see https://github.com/kubernetes/enhancements/pull/1111

Would appreciate if a workaround is issued for this asap. Thank you.

[ruifung]

Currently I think, the only workaround I've found (as a consumer of the chart) is to use a post-renderer to patch the annotation back in.

This is an excerpt from my HelmRelease definition (FluxCD) for this that patches the required annotation back in when it's deployed.

  postRenderers:
    - kustomize:
        patches:
          - patch: |
              - op: "add"
                path: "/metadata/annotations/api-approved.kubernetes.io"
                value: "https://github.com/kubernetes-sigs/external-dns/pull/2007"
            target:
              kind: CustomResourceDefinition
              name: dnsendpoints.externaldns.k8s.io

[mpechner-akasa]

Ignore this, I had to back port to 7.3.2. external-dns is broken. Would be interested in the tests and "works on my system" comments from the maintainers. I am on eks, k8s vers 1.24.

For me this was an upgrade from 7.5.5 - which had an issue with the CRD. The latest, 8.3.4.

I removed the CRD and applied the new one manually. I removed the helm chart setting to update CRD. External DNS is not bad since there is just one CRD to apply.

All good now.

The helm chart maintainer should handle this. But for me since other charts require I manually maintain the CRD, not an issue. What's one more.

[pinkfloydx33]

@ruifung thanks for that. I had been thinking about patching with Flux but wasn't sure how it would handle patching CRDs. It does the trick though!

[mpechner-akasa]

Seems to be fixed here: https://github.com/kubernetes-sigs/external-dns/pull/4488 I guess wait until bitnami updated teir chart, or we all just move to the external-dns chart.