ExternalConfig Array Merging Issue in Kubernetes vs Local Environment

[vineeth-au]

I've been troubleshooting a configuration merging issue for the past few days without success, so I'm hoping someone here can help.

Environment Details:

spring-boot-dependencies: 3.3.12 spring-cloud-starter-bootstrap: 4.1.4 spring-cloud-dependencies: 2023.0.3

Deployment: Kubernetes cluster

Configuration Setup: I'm using environment-specific YAML files: application.yaml, application-dev.yaml, application-staging.yaml, and application-prod.yaml.

application.yaml (base configuration):

my-application-name:
  action1:
    - "Item 1"
    - "Item 2"
    - "Item 3"
    - "Item 4"
    - "Item 5"

application-dev.yaml (environment override):

my-application-name:
  action1:
    - "Item 1"
    - "Item 6"
    - "Item 7"

The Problem:

When running locally, the configuration behaves as expected. However, when deployed to Kubernetes, the action1 array becomes: ["Item 1", "Item 6", "Item 7", "Item 4", "Item 5"].

This appears to be merging arrays rather than completely overriding them, but only in the Kubernetes environment. This is quite similar to the issue mentioned here.

[wind57]

can you provide a sample to re-produce this issue please? If I can see it in action, I can take a closer look on what is going on. thank you

[vineeth-au]

Fair enough. I saw this behaviour in our cluster, but unfortunately not when running locally. Give me couple of days and I'll try to make up a hello-world example (I'm thinking minikube) where this behaviour can be replicated. Is that ok?

[wind57]

that would be perfect

[vineeth-au]

Morning, @wind57 ... Hope you are doing good. Did not get a lot of time to work on this during the weekend and before I can submit any code I will need to check my company policies. Can I have until the end of this week to get back to you?

[wind57]

no worries at all, take your time

[vineeth-au]

Figured I'd give you an update. I was able to reproduce the issue on minikube!!! We are in the process of getting clearance from legal. Will keep you updated.

[vineeth-au]

Hey @wind57 , how are you doing? We got the all clear!!! Here is a hello-world example with the issue being reproduced. Let me know what you think. If you have any questions, feel free to ping me and we can sync up...

[vineeth-au]

FYI: We are using SpringBoot 3.3.12 (with compatible dependencies), however in this example I've used the latest version(3.5.0), for couple of reasons:

• To verify & validate this issue is present in the latest version.
• Make your life easier 😃 (hopefully)

[wind57]

Thank you. I will take a look next week, since Im on vacation.

[wind57]

I'm back from vacation, but have some pending things that I left, will do my best to look at this on Friday

[wind57]

@vineeth-au I've tried your sample just now, in kind. And here are my logs :

root@kind-control-plane:/# kubectl get pods -o wide
NAME                                  READY   STATUS    RESTARTS   AGE   IP           NODE                 NOMINATED NODE   READINESS GATES
spring-config-test-69b6554c6c-t8lbg   1/1     Running   0          10m   10.244.0.5   kind-control-plane   <none>           <none>
root@kind-control-plane:/# curl 10.244.0.5:8080
[Item 1, Item 6, Item 7]root@kind-control-plane:/#

And the logs show the same two profiles:

com.spring.Application                   : The following 2 profiles are active: "dev", "kubernetes"

So, I can't reproduce it.

[vineeth-au]

That is interesting... I'm on vacation and will be back next week. Can I get back to you then?

[vineeth-au]

Hey @wind57 , how are you doing? First of all thank you for taking the time to look into this... I just got back from vacation and finally got time to look into this. I think I was able to reproduce the issue you were facing. Would it be possible to catch up for 10 minutes when you are free and we could go over this together? Let me know what works, else I will document the steps that I did and we can take it from there.

[wind57]

There are a few things here that we need to look at.

• first one is this:

Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://10.96.0.1:443/api/v1/namespaces/default/configmaps. Message: configmaps is forbidden: User "system:serviceaccount:default:default" cannot list resource "configmaps" in API group "" in the namespace "default". Received status: Status(apiVersion=v1, code=403, details=StatusDetails(causes=[], group=null, kind=configmaps, name=null, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=configmaps is forbidden: User "system:serviceaccount:default:default" cannot list resource "configmaps" in API group "" in the namespace "default", metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Forbidden, status=Failure, additionalProperties={}).

And you get it, because the default account, does not have proper rights.

• "But I have created and configured a non-default account via the proper manifests". You did, but you did not link it to the deployment via:

kind: Deployment
apiVersion: apps/v1
metadata:
  name: spring-config-test
spec:
  template:
    spec:
      serviceAccountName: spring-config-test # this was not set

• Once, I set the serviceAccountName, I can re-produce your issue immediately.

• Yes, this is a bug on our side.

---

I don't know how to fix it it, but working on it.

@ryanjbaxter can you add the bug label here please?

[wind57]

sorry for the long delay, I am caught up in my actual work, so have little time. The good news, is that I wrote a unit tests where the problem reproduces and debugged and now know why it happens. I also debugged spring-boot and how they handle such cases, and I think there is a way for us to solve this. I'll give it a try this week

[vineeth-au]

Thank you for the update @wind57 ...

[wind57]

The PR is out for this one, here, but it will take Ryan or someone else from the team to take a closer look.

[wind57]

@vineeth-au thank you for the effort and the call we had where you explained the issue, that was very helpful. This is now fixed.

[vineeth-au]

Thank you for the update @wind57 ...