`skupper-network-observer` deployment stuck due to unavailability of `skupper-network-observer-client` secret

[vsomwanshi]

Describe the bug

While setting up the skupper-network-observer deployment get stuck in to containercreating state with event/error MountVolume.SetUp failed for volume "skupper-management-client" : secret "skupper-network-observer-client" not found. skupper-network-observer setup documentation link for reference.

Upon further looking at the yaml file seems that skupper-network-observer-client is a Certificate. I do not find skupper-network-observer-client is getting created as a secret in the configuration yaml.

# Source: network-observer/templates/client_certificate.yaml
apiVersion: skupper.io/v2alpha1
kind: Certificate
metadata:
  name: skupper-network-observer-client
  labels:
    app.kubernetes.io/name: network-observer
    app.kubernetes.io/instance: skupper-network-observer
    app.kubernetes.io/version: "2.1.0"
    app.kubernetes.io/part-of: skupper-network-observer
spec:
  ca: skupper-local-ca
  hosts:
  - skupper-router-local
  client: true
  subject: skupper-router-local

How To Reproduce Steps to reproduce the behavior:

Follow skupper-network-observer configuration yaml and deploy.

Expected behavior skupper-network-observer-client should be created as secret so skupper-network-observer deployment can refer secreName during deployment.

Environment details

• Skupper CLI: 2.1.0
• Skupper Operator (if applicable): [e.g. 1.5.0, 1.4.3]
• Platform: OpenShift

Additional context Add any other context about the problem here.

[c-kruse]

Hi @vsomwanshi

Do you have an active Site in the network-observer namespace? I would consider that a precondition for installing this chart with the default values. Perhaps that could be better documented somewhere?

The relevant helm configuration options: https://github.com/skupperproject/skupper/blob/2.1.2/charts/network-observer/values.yaml#L55-L72

# router configuration establishes the point at which the network observer attaches to the skupper network
router:
  endpoint: "amqps://skupper-router-local"
  certificate:
    create: true
    nameOverride: ""

router.endpoint points by default the network-observer at the "skupper-router-local" service created by default with each new Site.

router.certificate controls that that skupper-network-observer-client Certificate you noticed. When there's a Site, the controller should create a Secret with TLS creds with the same name. Without a Site, the "skupper-local-ca" Secret will not be in place to use to sign the new cert. I suspect the status of the client Certificate relays something to the effect of a "missing skupper-local-ca". Depending on your goals, we can probably look into finding a scheme that makes sense for you.

Ex: you could overwrite this setting router.certificate.create=false and router.certificate.nameOverride=myTLSCertSecretName. You'd just need a few things in place on the router/site side to make sure this all works together.

# This block configures the network-observer http service with TLS
tls:
  # secretName set to the secret containing the TLS certificate and key
  secretName: ""
  # openshiftIssued - configure annotations so that openshift will issue a
  # certificate
  openshiftIssued: false
  # skupperIssued configures the network-observer with a TLS certificate issued
  # by the skupper controller using the same mechanisim used to provision
  # certificates for linking sites.
  skupperIssued: true

The tls block points to a DIFFERENT set of certificates for the network-observer to use for its https server. I think they're better documented than the router block is, but thought I'd mention there's options on this side too.

[vsomwanshi]

Thanks @c-kruse I was able to deploy network observer successfully. I thought that deployment could be similar to the controller and we can deploy it under one namespace and scrape the data from all the sites from centralize namespace. But this is not the case we would need to install network observer in each site namespaces.

1. I have installed the network observer which up and running fine.

$ oc get pods
NAME                                        READY   STATUS      RESTARTS   AGE
skupper-network-observer-678f97cbcf-77djn   3/3     Running     0          97m
skupper-network-observer-setup-mx22k        0/1     Completed   0          96m
skupper-router-757d8b74f8-gptv7             2/2     Running     0          7d1h

2. I have created a service monitor so promethus can scrape the metrics from the defined ports of network observer service.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: skupper-network-observer
  namespace: sre-skupper
spec:
  endpoints:
    - interval: 30s
      port: https
      scheme: https
      tlsConfig:
        ca:
          secret:
            key: ca.crt
            name: skupper-network-observer-client
        cert:
          secret:
            key: tls.crt
            name: skupper-network-observer-tls
        insecureSkipVerify: true
        keySecret:
          key: tls.key
          name: skupper-network-observer-tls
  selector:
    matchLabels:
      app.kubernetes.io/name: network-observer

Somehow the metrics target is still down with below error, any idea about this ?

Even i've setup network policy explicitly as well;

kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: skupper-network-observer
spec:
  policyTypes:
  - Ingress

  podSelector:
    matchLabels:
      app.kubernetes.io/name: network-observer
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: sre-skupper
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: openshift-monitoring
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: openshift-user-workload-monitoring

    ports:
    - protocol: TCP
      port: https