Kustomization referencing manifest on GitHub fails to apply

[damoxc]

Describe the bug

Attempting to install the awx-operator using the Kustomize method, which references a manifest within the GitHub repository, this works fine when running kustomize build but committing it to the repository ready to be picked up by Flux results in:

✗ Kustomization reconciliation failed: kustomize build failed: accumulating resources: accumulation err='accumulating resources from './awx': read /tmp/kustomization-795774899/awx: is a directory': recursed accumulation of path '/tmp/kustomization-795774899/awx': accumulating resources: accumulation err='accumulating resources from 'github.com/ansible/awx-operator/config/default?ref=0.29.0': open /tmp/kustomization-795774899/awx/github.com/ansible/awx-operator/config/default?ref=0.29.0: no such file or directory': git cmd = '/usr/bin/git fetch --depth=1 origin 0.29.0': exit status 128

Steps to reproduce

1. Include a kustomization.yaml which references a file in a GitHub repository

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  # Find the latest tag here: https://github.com/ansible/awx-operator/releases
  - github.com/ansible/awx-operator/config/default?ref=<tag>

# Set the image tags to match the git version from above
images:
  - name: quay.io/ansible/awx-operator
    newTag: <tag>

# Specify a custom namespace in which to install AWX
namespace: awx

2. Attempt to apply

Expected behavior

Manifests would be fetched from GitHub, same behaviour as kustomize build.

Screenshots and recordings

No response

OS / Distro

Arch Linux

Flux version

v0.34.0

Flux check

► checking prerequisites ✔ Kubernetes 1.23.10+rke2r1 >=1.20.6-0 ► checking controllers ✔ helm-controller: deployment ready ► ghcr.io/fluxcd/helm-controller:v0.24.0 ✔ image-automation-controller: deployment ready ► ghcr.io/fluxcd/image-automation-controller:v0.25.0 ✔ image-reflector-controller: deployment ready ► ghcr.io/fluxcd/image-reflector-controller:v0.21.0 ✔ kustomize-controller: deployment ready ► ghcr.io/fluxcd/kustomize-controller:v0.28.0 ✔ notification-controller: deployment ready ► ghcr.io/fluxcd/notification-controller:v0.26.0 ✔ source-controller: deployment ready ► ghcr.io/fluxcd/source-controller:v0.29.0 ► checking crds ✔ alerts.notification.toolkit.fluxcd.io/v1beta1 ✔ buckets.source.toolkit.fluxcd.io/v1beta2 ✔ gitrepositories.source.toolkit.fluxcd.io/v1beta2 ✔ helmcharts.source.toolkit.fluxcd.io/v1beta2 ✔ helmreleases.helm.toolkit.fluxcd.io/v2beta1 ✔ helmrepositories.source.toolkit.fluxcd.io/v1beta2 ✔ imagepolicies.image.toolkit.fluxcd.io/v1beta1 ✔ imagerepositories.image.toolkit.fluxcd.io/v1beta1 ✔ imageupdateautomations.image.toolkit.fluxcd.io/v1beta1 ✔ kustomizations.kustomize.toolkit.fluxcd.io/v1beta2 ✔ ocirepositories.source.toolkit.fluxcd.io/v1beta2 ✔ providers.notification.toolkit.fluxcd.io/v1beta1 ✔ receivers.notification.toolkit.fluxcd.io/v1beta1 ✔ all checks passed

Git provider

No response

Container Registry provider

No response

Additional context

No response

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[stefanprodan]

Can you please try:

apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: GitRepository
metadata:
  name: awx-operator
  namespace: flux-system
spec:
  interval: 12h
  url: https://github.com/ansible/awx-operator
  ref:
    tag: 0.29.0
---
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: awx-operator
  namespace: flux-system
spec:
  interval: 1h
  retryInterval: 1m
  timeout: 5m
  path: ./config/default
  prune: true
  wait: true
  sourceRef:
    kind: GitRepository
    name: awx-operator
  targetNamespace: awx
  images:
    - name: quay.io/ansible/awx-operator
       newTag: 0.29.0

[damoxc]

Thanks a lot, that works as expected.

Is it mentioned in the documentation that fetching manifests from URLs isn't supported in kustomize.yaml? I did try to spot whether or not it was.

[stefanprodan]

Is it mentioned in the documentation that fetching manifests from URLs isn't supported in kustomize.yaml?

We do support remote bases but we encourage users to disable it. In your case, I suspect that cloning fails because Kustomize itself does not retry, nor does it cache the repo like Flux does when you use a GitRepository.

From docs:

For security and performance reasons, it is advised to disallow the usage of remote bases in Kustomize overlays. To enforce this setting, platform admins can use the --no-remote-bases=true controller flag.

[stefanprodan]

It may work better if you tell Kustomize to clone the repo over HTTPS like so:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - https://github.com/ansible/awx-operator/config/default?ref=<tag>

[polarroyo]

I am experiencing the same behavior, but only when the kustomize-controller runs on a ARM (aarch64) node. When it runs on a x86 node I do not have any issue.

kustomize-controller version that I can reproduce the issue:

• v0.33.0
• v0.32.0

[kifeo]

I have the same issue even with https:// prefix :

 ---
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 namespace: monitoring
 resources:
   - https://github.com/fluxcd/flux2/manifests/monitoring/monitoring-config?ref=v0.39.0

log :

accumulating resources: accumulation err='accumulating resources from 'https://github.com/fluxcd/flux2/manifests/monitoring/monitoring-config?ref=v0.39.0': URL is a git repository': git cmd = '/usr/bin/git fetch --depth=1 origin v0.39.0': exit status 128

[ilijamt]

I have a similar issue with this, the only difference is I'm building the CRDs with flux.

This is my Kustomization to build all the CRDs I need.

 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
   - github.com/ansible/awx-operator/config/crd?ref=1.1.3
   - external-secrets.io_clusterexternalsecrets.yaml
   - external-secrets.io_clustersecretstores.yaml
   - external-secrets.io_externalsecrets.yaml
   - external-secrets.io_secretstores.yaml
   - calico-crds.yaml
   - cert-manager.io_certificaterequests.yaml
   - cert-manager.io_certificates.yaml
   - cert-manager.io_challenges.acme.yaml
   - cert-manager.io_clusterissuers.yaml
   - cert-manager.io_issuers.yaml
   - cert-manager.io_orders.acme.yaml

When running both with version v4.5.7 and v5.1.1 this works and the CRDs for github.com/ansible/awx-operator/config/crd?ref=1.1.3 are present in the generated yaml.

kustomize build <kustomization.path> 

When running, which contains the kustomization above

flux build kustomization crds --path crds

the AWX crds are missing from the generated yaml.

I've tried with both github.com/ansible/awx-operator/config/crd?ref=1.1.3 and https://github.com/ansible/awx-operator/config/crd?ref=1.1.3

With github.com/ansible/awx-operator/config/crd?ref=1.1.3 I just get an error

✗ failed to generate kustomization.yaml: lstat /manifests/crds/github.com/ansible/awx-operator/config/crd/?ref=1.1.3: no such file or directory <nil> <nil>

and with https://github.com/ansible/awx-operator/config/crd?ref=1.1.3 the AWX CRDs are missing from the generated file.

$ flux check
► checking prerequisites
✔ Kubernetes 1.25.12-eks-2d98532 >=1.25.0-0
► checking controllers
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v0.36.0
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v1.1.0
✔ notification-controller: deployment ready
► ghcr.io/fluxcd/notification-controller:v1.1.0
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v1.1.0
► checking crds
✔ alerts.notification.toolkit.fluxcd.io/v1beta2
✔ buckets.source.toolkit.fluxcd.io/v1beta2
✔ gitrepositories.source.toolkit.fluxcd.io/v1
✔ helmcharts.source.toolkit.fluxcd.io/v1beta2
✔ helmreleases.helm.toolkit.fluxcd.io/v2beta1
✔ helmrepositories.source.toolkit.fluxcd.io/v1beta2
✔ kustomizations.kustomize.toolkit.fluxcd.io/v1
✔ ocirepositories.source.toolkit.fluxcd.io/v1beta2
✔ providers.notification.toolkit.fluxcd.io/v1beta2
✔ receivers.notification.toolkit.fluxcd.io/v1
✔ all checks passed