Partial service discovery when all-namespaces property is true

[Jimmy-Newtron]

Describe the bug I have created a simple REST micro-service that lists all discovered cluster services

I used the following dependency (this is the problem)

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-kubernetes-client-all</artifactId>
</dependency>

and the following property

spring:
  cloud:
   kubernetes: 
     discovery: 
       all-namespaces: true

Sample

ServiceController.java

import java.util.List;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.cloud.client.ServiceInstance;
import org.springframework.cloud.client.discovery.DiscoveryClient;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.ResponseBody;

@RestController
public class ServiceController {
  
  @Autowired
  DiscoveryClient discoveryClient;

  @GetMapping("/services")
	public Iterable<String> findAll() {
    return discoveryClient.getServices();
	}

  @GetMapping("/service/{serviceId}")
  @ResponseBody
	public List<ServiceInstance> getService(@PathVariable String serviceId) {
    return discoveryClient.getInstances(serviceId);
	}
}

MyApplication.java

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cloud.client.discovery.EnableDiscoveryClient;

@SpringBootApplication
@EnableDiscoveryClient
public class MyApplication {

  public static void main(String[] args) {
    SpringApplication.run(MyApplication.class, args);
  }
}

Workaround

When using the Fabric8 client the problem disappears

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-kubernetes-fabric8-all</artifactId>
</dependency>

In the documentation it is not clear if we need a ClusterRoleBinding or a simple RoleBinding to access all-namespaces

[wind57]

"the problem disappears", sorry exactly what problem are you talking about?

And second, the documentation is not going to explicitly say if you need a ClusterRoleBinding vs RoleBinding (do not confuse ClusterRole with Role which are entirely different things). In general, any well behaved organization is going to reject an install where a ClusterRoleBinding is present - this is dangerous, imo. For example we have tools in place that will reject such a manifest where a ClusterRoleBinding is requested. How you set-up your RBAC for this, is entirely up to your organization/product.

[LaSylv]

Having the same issue

The property spring.cloud.kubernetes.discovery.all-namespaces: true works with the fabric8 implementation, but not with the native one

[vksaamy]

Tested today and still the issue exists with spring-cloud-starter-kubernetes-discoveryclient version 2.1.4

[ryanjbaxter]

When I run this integration test, modified with

spring:
  cloud:
    kubernetes:
      discovery:
        all-namespaces: true

I get services from all namespaces.

https://github.com/spring-cloud/spring-cloud-kubernetes/tree/2.1.x/spring-cloud-kubernetes-integration-tests/discovery/kubernetes-client-discovery

[spring-cloud-issues]

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

[vksaamy]

I tried using this dependency

         <dependency>
		  <groupId>org.springframework.cloud</groupId>
		  <artifactId>spring-cloud-starter-kubernetes-discoveryclient</artifactId>
	</dependency>

and tried to list the services across namespaces using List<String> svclist = discoveryClient.getServices(); This svclist didn't have all the services from all namespaces but only had services from the same namespace where the calling service existed.

When using

              <dependency>
	    <groupId>org.springframework.cloud</groupId>
	    <artifactId>spring-cloud-starter-kubernetes-client-all</artifactId>
	</dependency>

with cluster role to query API server for services, I was able to get all services from all namespaces.

When using

         <dependency>
	    <groupId>org.springframework.cloud</groupId>
	    <artifactId>spring-cloud-starter-kubernetes-fabric8-all</artifactId>
	</dependency>

also with cluster role to query API server for services, I was able to get all services from all namespaces.

[ryanjbaxter]

Can you provide a complete, minimal, verifiable sample that reproduces the problem? It should be available as a GitHub (or similar) project or attached to this issue as a zip file.

[spring-cloud-issues]

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

[vksaamy]

I have created a gitrepo with my project and it can be run and the discovery client is not able to list services from other namespaces. The git repo is at https://github.com/vksaamy/Petclinc_spring_cloud_discovery_server The entire code was tested in Rancher Desktop.

[ryanjbaxter]

The only usage of discovery client i see in this sample is here https://github.com/vksaamy/Petclinc_spring_cloud_discovery_server/blob/a4979ef47d0b8bbffe39eece935550ba11ef5f72/PetClinic01/visits-svc1/src/main/java/org/springframework/samples/petclinic/visits/DiscoveryConfig.java#L37

And there isn't even a service for that.

Please simply the sample down to the bare minimum necessary to reproduce the problem.

[vksaamy]

The method that is used is getVet(Integer vetId) method which is trying to list all the services across all namespaces. It lists only two services - Visits and Discovery service. By right, it should display three services - Visits, Discovery ( in petclinic01 namespace) and Vetssvc2 (in petclinic02) namespace. The REST endpoint to invoke is http://<visit service IP>:9284/visits/1 . Pls. refer to the attached log captured.

[ryanjbaxter]

so you are using https://hub.docker.com/r/springcloud/spring-cloud-kubernetes-discoveryserver?

How are you configuring and deploying that?

[vksaamy]

Yes. from the docker image. The discover server is deployed through the yaml - discoveryserver_deployment.yml under petclinic01/K3S folder. The visitsvc bootstrap.yml point to this server through discovery-server-url: http://spring-cloud-kubernetes-discoveryserver.petclinic01.svc.cluster.local URL.

[ryanjbaxter]

I dont see where you are telling the discovery server to look across all namespaces by setting all-namespaces to true

[vksaamy]

It is configured in bootstrap.yml file of visits svc.

[ryanjbaxter]

Since the discovery server is supplying the service data you need to set all-namespaces to true on the discovery server not on the discovery client. The easiest way would be to set an environment variable in the deployment yaml of the discovery server.

[vksaamy]

Setting the env value at discovery server is able to fetch services from all namespaces. Thanks.

[vksaamy]

Now there is error as it can't query API server because i removed clusterrole for the visitssvc. The visitssvc is not supposed to query API server to discover service. It has to use discovery server to query service. So, should not need cluster role i believe. Besides, the discovery service is in the same namespace as visitissvc. Does loadbalanced RestTemplate works with DiscoveryClient and requests are load balanced across services instances from DiscoveryClient?

[ryanjbaxter]

You will need to remove this https://github.com/vksaamy/Petclinc_spring_cloud_discovery_server/blob/main/PetClinic01/visits-svc1/pom.xml#L127-L136

[vksaamy]

The reason the dependency was added was to load the config map. Pls. refer to the attached error.

[ryanjbaxter]

It looks like the InformerDiscoveryClient is still on the classpath, some dependency is pulling that in. Maybe do a maven tree command and look at which spring-cloud-kubernetes dependencies are on the classpath.

[vksaamy]

Don't see any InformerDiscoveryClient in the class path. Attached is the dependency tree from maven.

20221202.log

[ryanjbaxter]

Try setting kubernetes.informer.enabled=false

[vksaamy]

I tried this and still it is throwing the same exception if that dependency is not there.

[ryanjbaxter]

What permissions does the app have? I am not sure why it's happening. If you can provide a much simpler sample to reproduce I am happy to look at it further. The one above is much too complex for me to decipher why this is happening

[vksaamy]

Actually the problem is still there. When i updated the image pull policy to Always and when the dependency for discovery client took effect, the discovery of services were from the same namespace. This is irrespective of the cluster role assignment.

[ryanjbaxter]

There is not much more I can suggest until I have a simple reproducible sample to work with

[vksaamy]

I have updated the project removing codes not needed and updated the github project also with the steps to set up project and test the issue. I believe this setup of multiple namespace is needed to identify this issue.

[ryanjbaxter]

Once again there is no environment variable setting the all-namespace property to true in the deployment of the discovery server https://github.com/vksaamy/Petclinc_spring_cloud_discovery_server/blob/eed4c58e775dce2f388d311249a80b54e4786c89/PetClinic03/K3S/discoveryserver_deployment.yml#L67-L83

[vksaamy]

This env variable was already tested and not working. I added again and tested now. Still showing 1 service only. I have updated the changes to Github.