Publish HeartBeat Event For DiscoveryServer Based DiscoveryClient

[polifr]

Hi, I am using Spring Cloud 2021.0.1, and I am developing a Spring Gateway application for a k8s cluster. In this one, I already deployed the Spring Cloud Kuberneted DiscoveryServer using the yaml published in the documentation page (and using 2.1.1 version).

In the discovery client documentation is told that:

Spring Cloud Kubernetes can also watch the Kubernetes service catalog for changes and update the DiscoveryClient implementation accordingly. In order to enable this functionality you need to add @EnableScheduling on a configuration class in your application.

I did that:

@SpringBootApplication
@EnableDiscoveryClient
@EnableScheduling
public class GatewayApplication {

But when a new microservice is published in the cluster, it not catched in the routing tables. We managed to inspect what is read by the discovery client in the gateway by developing an additional RestController that inquiries the server through the ReactiveDiscoveryClient, like this:

@RestController
public class GatewayController {

    @Autowired
    private ReactiveDiscoveryClient client;
    
    @GetMapping("/services")
    public ResponseEntity<Mono<List<String>>> getServices() {
        try {
            return ResponseEntity.ok(this.client.getServices().collectList());
        } catch (Exception ex) {
            return ResponseEntity.internalServerError().build();
        }
    }
}

Opening the browser on localhost:8080/services, the whole list is given, even the "new" services (the ones that has been started AFTER the gateway). Restarting the gateway application (i.e. kubectl delete and apply) all the cluster services are reachable, meaning that che routing tables is fully populated.

I saw that in the old version (1.1.x) there was a KubernetesCatalogWatch, triggered by KubernetesCatalogWatchAutoConfiguration (and the relative yaml property, spring.cloud.kubernetes.discovery.catalog-services-watch.enabled, true by default), that had a @Scheduled method for catalg update. In the current version there is nothing similar: is the documentation still valid or there is another way to activate the automatic service discovery update?

Thanks in advance

[ryanjbaxter]

So is the problem that the service names are listed via the ReactiveDiscoveryClient but you cannot route requests to them via the gateway?

[polifr]

Yes; using this yaml configuration for the Spring Boot Gateway application:

spring:
  main:
    banner-mode: off
  application:
    name: "myapp-gateway"
  cloud:
    discovery:
      client:
        health-indicator:
          enabled: true
      enabled: true
    gateway:
      discovery:
        locator:
          enabled: true
          url-expression: "'http://'+serviceId+':8080'"
          lower-case-service-id: true
    kubernetes:
      discovery:
        enabled: true
        all-namespaces: false
        primary-port-name: "default-http"
        discovery-server-url: "http://myapp-discoveryserver"
        include-not-ready-addresses: true
      reload:
        enabled: true
        mode: polling
        period: 5000

And deploying the discovery server with a myapp-discoveryserver service id on the k8s cluster, I am able to contact the services using localhost:8080/service-name/service-apis after the gateway is started. But only the ones that are already "ready" on the cluster work with this logic; for the ones deployed after, I can check that are correctly extracted by the injected discovery client, using the additional rest controller, but the routing table is not updated. It's not a matter of update period - I waited hours for "something" to start and check out the new routing, but nothing occurred.

I really don't understand if this process relies still on some scheduled tasks, as written in the documentation, or there is some other (new) way to accomplish this.

[ryanjbaxter]

Can your try clearing the route cache and seeing that works after new services are added?

https://cloud.spring.io/spring-cloud-gateway/reference/html/#refreshing-the-route-cache

[polifr]

Enabling the actuator endpoint for refresh in the application.yml file like this:

management:
  endpoint:
    gateway:
      enabled: true
    health:
      probes:
        enabled: true
      show-details: always
    info:
      enabled: true
    prometheus:
      enabled: true
  endpoints:
    web:
      exposure:
        include:
        - gateway
        - health
        - info
        - prometheus

And doing a POST on the localhost:8080/actuator/gateway/refresh url, the routes are correctly updated and requests are forwarded also to the new services.

This means that the update process works, if there is an event that triggers the action. This endpoint has been designed to be inquired manually, or there is the way to put this into some automatic process managed by the discovery server?

[polifr]

Anyway, if this is the way to do this, perhaps the documentation may contain some references that aren't no more correct... I mean, the @EnableScheduling annotation referred before.

[ryanjbaxter]

There is a bit of confusion here....

@EnableScheduling is for the K8S API based discovery clients.

You are using the Spring Cloud Kubernetes Discovery Server, it makes a request to the server and the server is fetching the service info (much like Eureka). The @EnableScheduling annotation does not apply here.

The problem you are experiencing is that the routes in the Gateway are cached and nothing is telling the Gateway the services have changed.

One solution might be to implement some kind of heartbeat event https://github.com/spring-cloud/spring-cloud-gateway/blob/71014d10bfa040a08d3479b161b79b3c8cd0d4a8/spring-cloud-gateway-server/src/main/java/org/springframework/cloud/gateway/route/RouteRefreshListener.java

I am also wondering if the discovery server itself could publish a message to this effect and apps may react to that which are interested.

[polifr]

Thank you for the explanation; I'll give a try to that RouteRefreshListener, or something that, internally to the same gateway, may trigger the event (some kind of a self RestTemplate, at least).

[ryanjbaxter]

There are some suggested workaround here https://github.com/spring-cloud/spring-cloud-gateway/issues/1514

[wind57]

this is actually a very valid issue, we should absolutely align all 3 clients. k8s and fabric8 one already publish proper Heartbeat event.

This is in fact doable for the http based discovery client also, I'll propose a solution for this one soon. I still have pending PRs that I want to get a review on, but I really want this one in 3.0.6. Just FYI, its on my next TODO