spring-cloud-kubernetes icon indicating copy to clipboard operation
spring-cloud-kubernetes copied to clipboard

Config Client Integration Documentation for Spring Cloud Kubernetes Config Server

Open susimsek opened this issue 2 years ago • 9 comments

Hi, Can you add config client integration documentation for Spring Cloud Kubernetes Config Server.

I have 2 questions that I'm curious about this topic. Can you share code snippets related to these questions? thank you in advance:)

1.How can I fetch property values for different profiles on Spring Cloud Kubernetes Config Server? 2. How can I define property values for different profiles on Spring Cloud Kubernetes Config Server?

susimsek avatar Nov 19 '23 18:11 susimsek

Spring Cloud Kubernetes Config Server is just a Spring Cloud Config Server with an optional environment repository for Config Maps and Secrets. Fetching property values and separating property values is done just like any other config server.

https://docs.spring.io/spring-cloud-config/docs/4.0.4/reference/html

ryanjbaxter avatar Nov 20 '23 14:11 ryanjbaxter

i created sample configmap as follows for default,qa and prod profiles. i can access the prod or qa property values on the kubernetes config server but my app cannot access the prod and kubernetes property files on the config server.

I'm getting an error like following. database connection information is defined in the account k8s profile on the config server. But my app cannot access any properties on the config server for k8s and prod profiles.

Can you share config client sample for Spring Cloud Kubernetes Config Server? The reproduce example codes are shared via my Github, check here.
 Spring Cloud Config Server version: 3.0.4

App startup log image

App error log image

App k8s config on Spring Cloud Kubernetes Config Server image

App prod config on Spring Cloud Kubernetes Config Server image

App aplication.yml

spring:
  application:
    name: account
  profiles:
    active: k8s, prod
  config:
    import: configserver:http://localhost:8888/

App configmap

apiVersion: v1
kind: ConfigMap
metadata:
  name: account
  namespace: default
data:
  application.yml: |-
    build:
      version: "3.0"

    account:
      message: "Welcome to EazyBank account related local APIs "
      contactDetails:
        name: "John Doe - Developer"
        email: "[email protected]"
      onCallSupport:
        - (555) 555-1234
        - (555) 523-1345
    
    spring:
      main:
        allow-bean-definition-overriding: true
      output:
        ansi:
          enabled: always
    
    management:
      endpoints:
        web:
          exposure:
            include: "*"
      health:
        readiness-state:
          enabled: true
        liveness-state:
          enabled: true
      endpoint:
        shutdown:
          enabled: true
        health:
          probes:
            enabled: true
      info:
        env:
          enabled: true
      metrics:
        tags:
          application: ${spring.application.name}
      tracing:
        sampling:
          probability: 1.0
        enabled: true
    
    springdoc:
      oAuthFlow:
        tokenUrl: http://localhost:7080/realms/eazybank/protocol/openid-connect/token
    ---
    spring:
      config:
        activate:
          on-profile: qa
      main:
        allow-bean-definition-overriding: true
      output:
        ansi:
          enabled: always
    build:
      version: "2.0"

    account:
      message: "Welcome to EazyBank account related QA APIs "
      contactDetails:
        name: "Smitha Ray - QA Lead"
        email: "[email protected]"
      onCallSupport:
        - (666) 265-3765
        - (666) 734-8371

    management:
      endpoints:
        web:
          exposure:
            include: "*"
      health:
        readiness-state:
          enabled: true
        liveness-state:
          enabled: true
      endpoint:
        shutdown:
          enabled: true
        health:
          probes:
            enabled: true
      info:
        env:
          enabled: true
      metrics:
        tags:
          application: ${spring.application.name}
      tracing:
        sampling:
          probability: 1.0
        enabled: true
    ---
    spring:
      config:
        activate:
          on-profile: prod
      main:
        allow-bean-definition-overriding: true
      output:
        ansi:
          enabled: always
      devtools:
        restart:
          enabled: false
        livereload:
          enabled: false
    server:
      shutdown: graceful

    build:
      version: "1.0"

    account:
      message: "Welcome to EazyBank account related production APIs "
      contactDetails:
        name: "Reine Aishwarya - Product Owner"
        email: "{cipher}acfc59b7eb19990a084cf8e17024511670254f67d8b765bd2fda3538f85db0cf81a698183045520fd5e381e1d4efea66"
      onCallSupport:
        - (453) 392-4829
        - (236) 203-0384
      
    management:
      endpoints:
        web:
          exposure:
            include: "*"
      health:
        readiness-state:
          enabled: true
        liveness-state:
          enabled: true
      endpoint:
        shutdown:
          enabled: true
        health:
          probes:
            enabled: true
      info:
        env:
          enabled: true
      metrics:
        tags:
          application: ${spring.application.name}
      tracing:
        sampling:
          probability: 1.0
        enabled: true
    ---
    spring:
      config:
        activate:
          on-profile: k8s
      datasource:
        url: jdbc:postgresql://postgresql:5432/accountdb
        username: easybank
        password: "{cipher}9e37167bf33b458463119a7927279b4f53ea984472903a0c2b61197a8739714b"
        hikari:
          maximum-pool-size: 30
          minimum-idle: 1
          pool-name: Hikari
          auto-commit: false
      jpa:
        hibernate:
          ddl-auto: none
          naming:
            physical-strategy: org.hibernate.boot.model.naming.CamelCaseToUnderscoresNamingStrategy
            implicit-strategy: org.springframework.boot.orm.jpa.hibernate.SpringImplicitNamingStrategy
        show-sql: true
        open-in-view: false
        properties:
          hibernate.connection.provider_disables_autocommit: true
          hibernate.cache.redisson.fallback: true
          hibernate.cache.use_second_level_cache: true
          hibernate.cache.use_query_cache: true
          hibernate.generate_statistics: false
          hibernate.jdbc.batch_size: 25
          hibernate.order_inserts: true
          hibernate.order_updates: true
          hibernate.query.fail_on_pagination_over_collection_fetch: true
          hibernate.query.in_clause_parameter_padding: true
          jakarta.persistence.sharedCache.mode: ENABLE_SELECTIVE
          hibernate.cache.redisson.entity.expiration.max_entries: 10000
          hibernate.cache.redisson.entity.expiration.time_to_live: 3600000
          hibernate.cache.redisson.entity.expiration.max_idle_time: 1800000
          hibernate.cache.redisson.query.eviction.max_entries: 10000
          hibernate.cache.redisson.query.eviction.time_to_live: 3600000
          hibernate.cache.redisson.query.eviction.max_idle_time: 1800000
          org.hibernate.envers.audit_table_suffix: _history
          org.hibernate.envers.revision_field_name: revision
          org.hibernate.envers.revision_type_field_name: revision_type
      data:
        redis:
          host: redis-master
          port: 6379
          password: "{cipher}72f6a7d004cec8a7558dd0a94bd59b3b10205cfdad18d890d510e57ed17c5d7f44554bd293cdd529d1192202dad75c3d9c5ef970f16dab4e0c4d66f2df708b38"
      kafka:
        bootstrap-servers: http://kafka:9092
      cloud:
        kubernetes:
          discovery:
            all-namespaces: true
          discoveryServer:
            url: http://spring-cloud-kubernetes-discoveryserver:80/
            
    server:
      port: 8080

    eureka:
      client:
        enabled: false

    management:
      zipkin:
        tracing:
          endpoint: http://tempo-grafana-tempo-distributor:9411

susimsek avatar Nov 20 '23 19:11 susimsek

Your sample is way to complex to decipher or even reuse to reproduce the problem.

If you hit the /actuator/env endpoint when the k8s and prod profiles are active what is the result?

ryanjbaxter avatar Nov 20 '23 20:11 ryanjbaxter

i created simple project for config client integration. It seems that it only fetched the default profile on the spring cloud kubernetes config server . It did not fetch property values for prod and k8s environments. The reproduce example codes are shared via my Github, check here.

You can test the property values by typing the following command. Only default property values are loaded

curl --location 'http://localhost:8080/api/contact-info'

image

Actuator env api response

image

default profile on Spring Cloud Kubernetes Config Server

image.

k8s profile on Spring Cloud Kubernetes Config Server

image

prod profile on Spring Cloud Kubernetes Config Server image

susimsek avatar Nov 21 '23 11:11 susimsek

https://github.com/susimsek/spring-cloud-kubernetes-config-server-sample

This returns a 404

ryanjbaxter avatar Nov 21 '23 14:11 ryanjbaxter

i changed project visibility.it is currently public.

susimsek avatar Nov 21 '23 14:11 susimsek

Thanks this helps. There are actually 2 problems that this illustrates.

First the kubernetes config server will return property sources from the same config map with the same property source name. For example, when the config client requests configuration data without any profiles you will get a response like

{
  "name": "account",
  "profiles": [
    "default"
  ],
  "label": null,
  "version": null,
  "state": null,
  "propertySources": [
    {
      "name": "configmap.account.default",
      "source": {
        "build.version": "3.0",
        "account.message": "Welcome to EazyBank account related local APIs ",
        "account.contactDetails.email": "[email protected]",
        "account.onCallSupport[1]": "(555) 523-1345",
        "account.onCallSupport[0]": "(555) 555-1234",
        "account.contactDetails.name": "John Doe - Developer"
      }
    }
  ]
}

Then the config client will make a second request with all active profiles (in you example k8s and prod) and it will return

{
  "name": "account",
  "profiles": [
    "k8s,prod"
  ],
  "label": null,
  "version": null,
  "state": null,
  "propertySources": [
    {
      "name": "configmap.account.default",
      "source": {
        "build.version": "1.0",
        "account.message": "Welcome to EazyBank account related kubernetes APIs ",
        "account.contactDetails.email": "[email protected]",
        "account.onCallSupport[1]": "(236) 203-0384",
        "account.onCallSupport[0]": "(453) 392-4829",
        "spring.config.activate.on-profile": "k8s",
        "account.contactDetails.name": "Reine Aishwarya - Product Owner"
      }
    }
  ]
}

Since Spring Boot is keeping track of all the property sources that we add to the environment and both property sources get added with the same name the one without any active profiles overrides the k8s and prod profile one. We can fix this by making the property source names unique (for example by appending any active profiles).

Second problem relates the the response from the config server when there are active profiles.

What we should be returning is all the property sources for all active profiles, so in your example I would expect to see the prod and k8s and default property sources returned and then the client will resolve which one takes priority based on the order they are returned.

This is going to take some digging in order to figure out what the right solution is.

ryanjbaxter avatar Nov 30 '23 20:11 ryanjbaxter

just FYI, next year I plan to start digging into config-server a lot more and potentially will reach this bug also

wind57 avatar Nov 30 '23 20:11 wind57

I am going to dig into this, I am already down the rabbit hole

ryanjbaxter avatar Nov 30 '23 20:11 ryanjbaxter