Ingress example not working with current NetworkPolicies

[paulfantom]

What happened?

After migrating to newest version (commit https://github.com/prometheus-operator/kube-prometheus/commit/73bbec9659941e95301d684bd54d39b9644b21b5) I lost all ingress connectivity to grafana, alertmanager, and prometheus.

Did you expect to see some different?

Services are accessible when used with Ingress example

How to reproduce it (as minimally and precisely as possible):

Run stack with Ingress example

Environment

• Prometheus Operator version:

  Insert image tag or Git SHA here

• Kubernetes version information:

  kubectl version

• Kubernetes cluster kind:

  insert how you created your cluster: kops, bootkube, tectonic-installer, etc.

• Manifests:

insert manifests relevant to the issue

• Prometheus Operator Logs:

Insert Prometheus Operator logs relevant to the issue here

• Prometheus Logs:

Insert Prometheus logs relevant to the issue here

Anything else we need to know?:

[paulfantom]

Additionally seems like grafana cannot reach prometheus

[ArthurSens]

Additionally seems like grafana cannot reach prometheus

We have a community contribution attempting to fix this at https://github.com/prometheus-operator/kube-prometheus/pull/1710, but I agree that it's a quite serious bug so we might might want to fix it quickly 😬

[sidharthv96]

Adding (import 'kube-prometheus/addons/networkpolicies-disabled.libsonnet') does not seem to be disabling the networkpolicies. Is there something I'm missing?

local kp =
	(import 'kube-prometheus/main.libsonnet') +
        (import 'kube-prometheus/addons/all-namespaces.libsonnet') + 
        (import 'kube-prometheus/addons/networkpolicies-disabled.libsonnet') +
	(import './config.libsonnet');

[kquinsland]

Adding (import 'kube-prometheus/addons/networkpolicies-disabled.libsonnet') does not seem to be disabling the networkpolicies. Is there something I'm missing?

local kp =
	(import 'kube-prometheus/main.libsonnet') +
        (import 'kube-prometheus/addons/all-namespaces.libsonnet') + 
        (import 'kube-prometheus/addons/networkpolicies-disabled.libsonnet') +
	(import './config.libsonnet');

This does work for me, now.

[AlesKrajnik]

The latest release v0.11 introduces restrictive network policies that do not allow access to any of the pods from anywhere but internal use. This results in timeouts (HTTP 504) from load balancers.

One solution is to disable all network policies as shown above. Alternatively (and arguably safer), suitable network policies can be added to allow needed access.

In my case, I want to expose Prometheus, AlertManager and Grafana from load balancers - pods ingress-nginx in the namespace ingress-nginx. (Load balancers are restricted to be accessible only from VPN.) And I want the NetworkPolicy configuration to be part of the main Jsonnet file so that the network policies are generated together with the rest of the kube-prometheus stack.

The following configuration in Jsonnet does the trick (example redacted to contain only relevant configuration):

local kp =
    (import 'kube-prometheus/main.libsonnet') +

    // ... some other configuration here

    alertmanager+: {
        networkPolicy+: {
            spec+: {
                ingress+: [{
                    from: [{
                        podSelector: {
                            matchLabels: {
                                'app.kubernetes.io/name': 'ingress-nginx',
                            },
                        },
                        namespaceSelector: {
                            matchLabels: {
                                'kubernetes.io/metadata.name': 'ingress-nginx',
                            },
                        },
                    }],
                    ports: [{
                        port: 'web',
                        protocol: 'TCP',
                    }],
                }],
            },
        },
    },

    prometheus+: {
        networkPolicy+: {
            spec+: {
                ingress+: [{
                    from: [{
                        podSelector: {
                            matchLabels: {
                                'app.kubernetes.io/name': 'ingress-nginx',
                            },
                        },
                        namespaceSelector: {
                            matchLabels: {
                                'kubernetes.io/metadata.name': 'ingress-nginx',
                            },
                        },
                    }],
                    ports: [{
                        port: 'web',
                        protocol: 'TCP',
                    }],
                }],
            },
        },
    },

    grafana+: {
        networkPolicy+: {
            spec+: {
                ingress+: [{
                    from: [{
                        podSelector: {
                            matchLabels: {
                                'app.kubernetes.io/name': 'ingress-nginx',
                            },
                        },
                        namespaceSelector: {
                            matchLabels: {
                                'kubernetes.io/metadata.name': 'ingress-nginx',
                            },
                        },
                    }],
                    ports: [{
                        port: 'http',
                        protocol: 'TCP',
                    }],
                }],
            },
        },
    };