configMapGenerator: cannot merge or replace from components

[sgurdiel]

What happened?

Build fails

What did you expect to happen?

Build succeeds and configMap in overlay is merged with component configMap

How can we reproduce it (as minimally and precisely as possible)?

# components/co1/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component

configMapGenerator:
- name: configmap-name
  namespace: default
  literals:
  - hello=world

# overlays/o1/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

components:
- ../../components/co1

configMapGenerator:
- name: configmap-name
  namespace: default
  behavior: merge
  literals:
  - hello=planet

kustomize build overlays/o1

Expected output

apiVersion: v1
data:
  hello: planet
kind: ConfigMap
metadata:
  name: configmap-name-m52bmf6b5t
  namespace: default

Actual output

Error: merging from generator &{0xc000390270 <nil>}: id resid.ResId{Gvk:resid.Gvk{Group:"", Version:"v1", Kind:"ConfigMap", isClusterScoped:false}, Name:"configmap-name", Namespace:"default"} does not exist; cannot merge or replace

Kustomize version

5.6.0, 5.5.0

Operating system

Linux

[k8s-ci-robot]

This issue is currently awaiting triage.

SIG CLI takes a lead on issue triage for this repo, but any Kubernetes member can accept issues by applying the triage/accepted label.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[sgurdiel]

In case this might help troubleshoot...

Replacing the behavior from merge to create in overlays/o1/kustomization.yml produces this output:

Error: merging from generator &{0xc0002b5520 <nil>}: id resid.ResId{Gvk:resid.Gvk{Group:"", Version:"v1", Kind:"ConfigMap", isClusterScoped:false}, Name:"configmap-name", Namespace:"default"} exists; behavior must be merge or replace

[adoramshoval]

Hi, can I work on this?

[sda399]

Hi

components are expected to execute after reading resources and adding generators ,before applying transformers and validation

Your generator is running before processing the component. You cannot merge in non existing CM

[sgurdiel]

Use case: Allowing overlays override (merge) configMaps created by reusable pieces of configuration logic (components).

Example: A database component that creates a configMap with database configuration settings and 2 different environments defined as overlays that include this component but require distinct database configuration setting.

This stopped working with v5 major release and I do not recall this being announced as an intended breaking change.

[sda399]

v5.1.0 released 2 years ago, the change is documented in the release notes

processing order is not so obvious topic, I guess.

anyway, for your use case, you still can patch your configmap in your overlays

#! /bin/sh

R=root
[ ! -d $R ] || rm -r $R
mk() { mkdir -p ${1%/*}  && echo "$2" >$1 ; }

mk $R/components/co1/kustomization.yaml '
# components/co1/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component

configMapGenerator:
- name: configmap-name
  namespace: default
  literals:
  - hello=world
'
mk $R/overlays/o1/kustomization.yaml '
# overlays/o1/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

components:
- ../../components/co1

patches:
- patch: |-
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: configmap-name
      namespace: default
    data:
      hello: planet
'

#---
cat <<EOF > expected.yaml
apiVersion: v1
data:
  hello: planet
kind: ConfigMap
metadata:
  name: configmap-name-m52bmf6b5t
  namespace: default
EOF
#---
kustomize build --stack-trace $R/overlays/o1/ > result.yaml
printf "%-64s%s\n" expected.yaml result.yaml
printf "%130s\n" " " | tr ' ' '-'
diff -W130 -y expected.yaml result.yaml

[Elegant996]

@adoramshoval Confirming the same issue. Will use snippets here but the issue should be clear:

base.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

configMapGenerator:
  - name: app-env
    envs:
      - .env

extras.yaml:

apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component

configMapGenerator:
  - name: app-env
    behavior: merge
    envs:
      - .env

We get an error: Error: merging from generator &{0xc00046a340 <nil>}: id resid.ResId{Gvk:resid.Gvk{Group:"", Version:"v1", Kind:"ConfigMap", isClusterScoped:false}, Name:"app-env", Namespace:""} does not exist; cannot merge or replace

Okay, let's try that again without the behavior:

extras.yaml:

apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component

configMapGenerator:
  - name: app-env
    # behavior: merge
    envs:
      - .env

Now we get a complete contradiction: Error: accumulating components: accumulateDirectory: "recursed accumulation of path '/home/runner/work/test/failing-components': merging from generator &{0xc00038d860 <nil>}: id resid.ResId{Gvk:resid.Gvk{Group:\"\", Version:\"v1\", Kind:\"ConfigMap\", isClusterScoped:false}, Name:\"app-env\", Namespace:\"\"} exists; behavior must be merge or replace"

No matter how you slice it, there's an issue here. You can't state the configmap doesn't exist then complain that it does.

[adoramshoval]

Hi all! Thank you for letting me work on this issue. I am new to the code base so it might take me some time 🤗

[adoramshoval]

/assign

[adoramshoval]

It seems like everything lies within the kusttarget.go file which controls the accumalation of resources. After some digging, it does not seem like there is a problem, it looks like everything does what is supposed to do. (TLDR at the end)

For merge or replace behavior, the chain of events goes as follows:

1. The KustTarget.accumulateTarget is executed: https://github.com/kubernetes-sigs/kustomize/blob/7c04cbb23791c0675cfaa3806ef9b99d59698a98/api/internal/target/kusttarget.go#L197-L199
2. Then, runGenerators is executed: https://github.com/kubernetes-sigs/kustomize/blob/7c04cbb23791c0675cfaa3806ef9b99d59698a98/api/internal/target/kusttarget.go#L225-L228
3. Which, at the end, goes through each generated resource map and attempts to make the resource accumulator absorb it: https://github.com/kubernetes-sigs/kustomize/blob/7c04cbb23791c0675cfaa3806ef9b99d59698a98/api/internal/target/kusttarget.go#L291-L294
4. The absorbtion is done by going through each resource in the absorbed resource map and attempting to append, replace or merge the resource, which correlates to the resource's chosen behavior - create, merge or replace: https://github.com/kubernetes-sigs/kustomize/blob/7c04cbb23791c0675cfaa3806ef9b99d59698a98/api/resmap/reswrangler.go#L560

As can be seen in the appendReplaceOrMerge function, if no older version of the resource you are currently running the function on exists in the main resource map, you can't use merge or replace behaviors, which is expected, as there is no resource to merge with nor to be replaced. https://github.com/kubernetes-sigs/kustomize/blob/7c04cbb23791c0675cfaa3806ef9b99d59698a98/api/resmap/reswrangler.go#L562-L568

In the case of create behavior, the runGenerators mentioned above will run correctly and create the ConfigMap as expected, the issue will come down when the components are processed, which is right after the runGenerators, here: https://github.com/kubernetes-sigs/kustomize/blob/7c04cbb23791c0675cfaa3806ef9b99d59698a98/api/internal/target/kusttarget.go#L230-L235

1. This function in turn create a KustTarget for each component directory and attempts to accumulate that directory, the same way the original kustomization directory was accumulated: https://github.com/kubernetes-sigs/kustomize/blob/7c04cbb23791c0675cfaa3806ef9b99d59698a98/api/internal/target/kusttarget.go#L472-L483
2. There, accumulateTarget is executed for the component directory, which in turn executes the runGenerators again: https://github.com/kubernetes-sigs/kustomize/blob/7c04cbb23791c0675cfaa3806ef9b99d59698a98/api/internal/target/kusttarget.go#L519-L522
3. Now, when the runGenerators function is executed, there is already a resource which has the same ID as the resource being generated in the component, which means the following case will be entered: https://github.com/kubernetes-sigs/kustomize/blob/7c04cbb23791c0675cfaa3806ef9b99d59698a98/api/resmap/reswrangler.go#L573-L601

https://github.com/kubernetes-sigs/kustomize/blob/7c04cbb23791c0675cfaa3806ef9b99d59698a98/api/resource/resource.go#L397-L403

4. Here, since no behavior was specified in the component itself, an empty behaior be will returned, which will enter the default case and raise the error you are getting.

• TLDR for merge or replace behaviors inside the kustomization.yaml configMapGenerator: This is the correct behavior, you are unable to merge or replace the generated config map with non existing config map. The generator inside the kustomization.yaml file is run before the component one, which means there are no resources to merge or replace with.
• TLDR for create behavior inside the kustomization.yaml configMapGenerator: By specifying a behavior, merge or replace at the component configMapGenerator you could avoid the error you are getting, otherwise, this seems like the correct behavior.

Continuation:

What we could take from this is making the raised error more understandable to the user, as from the user's POV the resource does not exist, which is confusing. Maybe we could add a case for create behavior at the component level, which will raise this error, and another case for an unspecified behavior which will have an understandable reasoning for the failure. I will create a PR for that.

Let me know if I missed anything here, or if I am completely wrong.

[adoramshoval]

@sgurdiel , @Elegant996 , @sda399 Was that a sufficient answer? Can we close this issue?

[Elegant996]

@adoramshoval In the example I listed though, the Kustomization would create the configmap ahead of processing Components, correct? In this instance, merge should work. This is also outlined in the components documentation.

Perhaps I'm misunderstanding, but the example https://github.com/kubernetes-sigs/kustomize/issues/5924#issuecomment-3187058202 shows that the generator with both a create and merge scenario in components.

In the scenario (specific to the Kind: Component), create states that the configmap already exists (it does) and merge states that it does not exist (it does). Both cannot be true simultaneously.

Per your comment, this would indicate an issue during step 3, when behavior: merge, likely this portion:

https://github.com/kubernetes-sigs/kustomize/blob/7c04cbb23791c0675cfaa3806ef9b99d59698a98/api/resmap/reswrangler.go#L560-L572

More specifically, we are not getting a match for a resource that does exist. Hence Error: merging from generator &{0xc00046a340 <nil>}: id resid.ResId{Gvk:resid.Gvk{Group:"", Version:"v1", Kind:"ConfigMap", isClusterScoped:false}, Name:"app-env", Namespace:""} does not exist; cannot merge or replace. This would imply that matches is 0 which is incorrect since the configmap does exist.

Now when behavior: create (again for Kind: Component), suddenly the resource is found bringing matches to 1:

https://github.com/kubernetes-sigs/kustomize/blob/7c04cbb23791c0675cfaa3806ef9b99d59698a98/api/resmap/reswrangler.go#L573-L601

This is confirmed with Error: accumulating components: accumulateDirectory: "recursed accumulation of path '/home/runner/work/test/failing-components': merging from generator &{0xc00038d860 <nil>}: id resid.ResId{Gvk:resid.Gvk{Group:\"\", Version:\"v1\", Kind:\"ConfigMap\", isClusterScoped:false}, Name:\"app-env\", Namespace:\"\"} exists; behavior must be merge or replace". Somehow, behavior is influencing the matching.

[adoramshoval]

Hi @Elegant996 , thanks for your response. I am trying to reproduce your issue but I am not getting the same results. Could you confirm the following is your setup:

# overlays/o1/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

components:
  - ../../components/co1/
configMapGenerator:
  - name: app-env
    envs:
      - .env

# components/co1/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component

configMapGenerator:
  - name: app-env
    behavior: merge
    envs:
      - .env

and,

# components/co1/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component

configMapGenerator:
  - name: app-env
    # behavior: merge
    envs:
      - .env

[Elegant996]

Confirmed. Please note that this is a single component. We're just commenting out behavior for testing purposes (since it said it didn't exist). Are you succeeding in merging?

[adoramshoval]

Confirmed. Please note that this is a single component. We're just commenting out behavior for testing purposes (since it said it didn't exist). Are you succeeding in merging?

Yes, merging works as expected for me. As explained in my previous comment, generators are accumulated before components, therefore, merging behavior on the component works as expected. We are entering this case: https://github.com/kubernetes-sigs/kustomize/blob/7c04cbb23791c0675cfaa3806ef9b99d59698a98/api/resmap/reswrangler.go#L573-L601

When removing the behavior, we are essentially using the unspecified behavior, which is not merge or replace. Since generators run before components, the generated config map is created, and then the component is accumulated, and since we not using merge or replace behavior, and error is raised with the exists text, which is refering to the generated config map from the first generator.

Error: accumulating components: accumulateDirectory: "recursed accumulation of path '/home/cloud-user/kustomize/test-kustomize/components/co1': merging from generator &{0xc0003da0d0 <nil>}: id resid.ResId{Gvk:resid.Gvk{Group:\"\", Version:\"v1\", Kind:\"ConfigMap\", isClusterScoped:false}, Name:\"app-env\", Namespace:\"\"} exists; behavior must be merge or replace"

[Elegant996]

@adoramshoval This is exactly as described; perhaps its the accumulation of resources through another kustomization? This shouldn't matter though; all directories should be traversed before generators. Components should apply after that.

In this case, ../base would contain the kustomization.yaml that defines generator.

# ../overlay/kustomization.yaml
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - ../base

components:
  - ../components/co1

# ../base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

configMapGenerator:
  - name: app-env
    envs:
      - .env

# ../components/co1/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component

configMapGenerator:
  - name: app-env
    behavior: merge
    envs:
      - .env

The ordering and explanation makes perfect sense but the results contradict that. Even the OP had the same bug. The question is why does it not hit case: 1 for us when using behavior: merge. -

[sda399]

@Elegant996 hi, imho, this issue is growing and diverging from the original request. like I commented above, this is an order of execution problem. An order that was amended in a certain version and produced an error in OP's use case.

It would probably be more beneficial, to open a github Discussion or a new Issue with a precise use case as a new base of discussion.