Support running cortex inside istio service mesh

[ghost]

Distributor is not able to deliver data to ingester (as its speaking with ingesters directly and hostname is not set) Is there any knowledge base how Run cortex in istio service mesh ?

Environment:

• Infrastructure: EKS (s3 + dynamodb)
• Deployment tool: helm

Thank you,

[pracucci]

I'm not familiar with Istio or service meshes in general, so I'm just going to share some details about how Cortex works.

Distributors, as well as queriers or rulers, need to be able to directly connect to ingesters discovered via the hash ring. The ingesters IP address exposed in the ring must be directly addressable by other Cortex components, but you can control it via the -ingester.lifecycler.addr CLI option (set it to the IP address of a specific ingester).

[pstibrany]

Note that we use the ring, and thus direct connection to IPs in some other places as well:

• queriers connecting to store-gateways
• rulers connecting to each other

I'm not aware of anyone using Cortex with Istio.

[simonxmh]

+1 we are also running into issues running Cortex with Istio, especially running with the mTLS. Distributor -> ingester , query-frontend -> queriers are both restricted from communicating.

[stale[bot]]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.

[snuggie12]

I'm attempting to get loki in microservices mode working and seeing the same issues you all are mentioning. I think it could work if you could tell the distributor or querier to use a DNS entry instead of an IP address. For example, there currently exists loki-ingester-0.loki-ingester-headless.monitoring.svc.cluster.local in my installation. You can then create a ServiceEntry tracking those to make it work.

[pracucci]

I think it could work if you could tell the distributor or querier to use a DNS entry instead of an IP address.

Cortex (as well as Loki) services need to directly address ingesters, due to how sharding and replication works. For this reason, we have a shared hash ring where ingesters are registered by IP and other services (ie. distributors, queriers, rulers) directly address ingesters by IP. Direct addressing is required and something I see it difficult to move away.

That being said, to my understand it could work to you if each ingester would be addressed by a per-pod hostname instead of IP. Is my understanding correct?

[snuggie12]

I believe so. This is how something like zookeeper or kafka works:

apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
  name: ingester
spec:
  hosts:
  - ingester-0.ingester-headless.monitoring.svc.cluster.local
  - ingester-1.ingester-headless.monitoring.svc.cluster.local
  - ingester-2.ingester-headless.monitoring.svc.cluster.local
  ports:
  - name: grpc
    number: 9095
    protocol: TCP

[dsabsay]

The solution I've arrived at is:

• Create a custom controller that creates k8s Services for each Pod that is accessed directly via IP (e.g. ingesters, store-gateways, rulers, etc.). I started with this example and modified as needed. Critically, the generated service name is a simple pattern that uses the IP address.
• Create a custom EnvoyFilter in Istio (I used Lua) which rewrites the :authority header of outbound requests, changing it from an IP address to the service name. This filter needs to be applied to all pods that directly access pods via IP, which is not necessarily the same pods that are accessed by IP. Off the top of my head, you need distributors, rules, queriers, and maybe some others. If you use Kiali, it makes it easy to see which paths aren't working. Examples/reference for writing Lua filters here.

This solution has not been battle-tested yet, but appears to work in a small, low-traffic environment.

[snuggie12]

Is there any chance that cortex/loki will adopt file SD like thanos? https://github.com/thanos-io/thanos/blob/main/docs/service-discovery.md#file-service-discovery

One small but important detail that I don't know about is does SD look at DNS names, resolve them, and then speak to IPs or does it speak to DNS names?

[bboreham]

Cortex SD will resolve a DNS name into multiple targets, which makes it suitable for use with a Kubernetes "headless service", or any other mechanism which gives the same kind of result.

We would consider a proposal to adopt the Thanos SD mechanism, but best to write a bit more detail about how it would be done before embarking on an implementation.

[snuggie12]

Cortex SD will resolve a DNS name into multiple targets, which makes it suitable for use with a Kubernetes "headless service", or any other mechanism which gives the same kind of result.

Yeah, headless services work. Though we should be specific with the word "resolve". If it's pre-resolving and then using IPs it won't work, but if it initiates a connection using a DNS name and normal DNS resolution happens then it will work.

We would consider a proposal to adopt the Thanos SD mechanism, but best to write a bit more detail about how it would be done before embarking on an implementation.

I thought you both shared code sometimes so I was referring to an actual copy of it, but essentially there's just a file format they support and the onus is on the user to get the files in place. After that it is the usual watching for a file change and reloading since it's usually done with a ConfigMap or Secret.

[bboreham]

we should be specific with the word "resolve"

There are multiple cases:

• calling via the "ring", for ingesters, store-gateway: by IP address as described earler in this issue.
• calling memcached from query-frontend: SRV records are looked up in DNS, then what comes back is passed to Dial as-is.
• calling memcached elsewhere: optional whether the results of SRV lookup are immediately resolved into IPs (dnssrv) or not (dnssrvnoa).
• queriers calling query-frontend: SRV lookup immediately resolved into IPs.

So a proposal which went into some detail about which would be changed and how, would help to flush any complexities. E.g. should the last three be unified?

[snuggie12]

@dsabsay I should have thought about something like metacontroller, good call! I went a slightly different path though. Mine uses a CompositeController with a CRD I created called DynamicServiceEntry which creates a ServiceEntry and the addresses field is dynamically updated with the pod IPs of pods matching your label. In theory this should work with resolution: NONE but I have yet to test.

@bboreham Thank you, this breakdown is helpful because originally I was only concerned with the ring, but couldn't figure out why other pieces weren't working either.

As for proposal:

• I think it's fair to say the ring is off the table. I'll continue working on the metacontroller I listed above, but it's an academic exercise. Other users can simply use two annotations on the pod to get that working: traffic.sidecar.istio.io/exclude{In,Out}boundPorts: 7946.
• For the last 3, the services needs some unified way to call each other via a DNS name. If that is dnssrvnoa then that is fine. To make sure I understand what you're saying above, it would specifically need:

# works with istio
curl www.foo.com

# does NOT work with istio
ip=$(dig +short www.foo.com) # resolves to a single IP in this example
curl $ip

[batistein]

Does anyone have a working example they can share? I would be very grateful for any help.

[kendrickclark]

The solution I've arrived at is:

• Create a custom controller that creates k8s Services for each Pod that is accessed directly via IP (e.g. ingesters, store-gateways, rulers, etc.). I started with this example and modified as needed. Critically, the generated service name is a simple pattern that uses the IP address.
• Create a custom EnvoyFilter in Istio (I used Lua) which rewrites the :authority header of outbound requests, changing it from an IP address to the service name. This filter needs to be applied to all pods that directly access pods via IP, which is not necessarily the same pods that are accessed by IP. Off the top of my head, you need distributors, rules, queriers, and maybe some others. If you use Kiali, it makes it easy to see which paths aren't working. Examples/reference for writing Lua filters here.

This solution has not been battle-tested yet, but appears to work in a small, low-traffic environment.

The solution I've arrived at is:

• Create a custom controller that creates k8s Services for each Pod that is accessed directly via IP (e.g. ingesters, store-gateways, rulers, etc.). I started with this example and modified as needed. Critically, the generated service name is a simple pattern that uses the IP address.
• Create a custom EnvoyFilter in Istio (I used Lua) which rewrites the :authority header of outbound requests, changing it from an IP address to the service name. This filter needs to be applied to all pods that directly access pods via IP, which is not necessarily the same pods that are accessed by IP. Off the top of my head, you need distributors, rules, queriers, and maybe some others. If you use Kiali, it makes it easy to see which paths aren't working. Examples/reference for writing Lua filters here.

This solution has not been battle-tested yet, but appears to work in a small, low-traffic environment.

@snuggie12 Do you have an example of the EnvoyFilter that you used?

[dsabsay]

Here is a filter I wrote in Lua. The workloadSelector selectively applies this filter to certain Pods. The outbound requests go to e.g. service-per-pod-123-123-123-123.cortex.svc.cluster.local:9095.

Then you need to manually or automatically create DNS entries for each Pod that must be accessed in this way. I did this with metacontroller as mentioned previously. There are multiple ways to do it. If you are creating Service objects, the query-frontend service needs publishNotReadyAddresses: true because query-frontend isn't considered ready until queriers connect (this may no longer be required; I'm not sure).

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: ip-to-service-rewrite
  namespace: cortex
spec:
  configPatches:
  - applyTo: HTTP_FILTER
    match:
      context: SIDECAR_OUTBOUND
      listener:
        filterChain:
          filter:
            name: envoy.filters.network.http_connection_manager
            subFilter:
              name: envoy.filters.http.router
        portNumber: 9095
    patch:
      operation: INSERT_BEFORE
      value:
        name: envoy.lua
        typed_config:
          '@type': type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua
          inlineCode: |
            -- http://www.lua.org/manual/5.1/manual.html#5.4.1
            function is_ip_and_port(s)
              if s:match("^%d+%.%d+%.%d+%.%d+:%d+$") == nil then
                return false
              else
                return true
              end
            end

            -- all outbound requests go through this function
            function envoy_on_request(request_handle)
              request_handle:logInfo("authority: "..request_handle:headers():get(":authority"))
              local authority = request_handle:headers():get(":authority")
              if is_ip_and_port(authority) == false then
                request_handle:logInfo("skipping. authority is not an IP-port")
                do return end
              end

              local s, e = string.find(authority, ":")
              if s == nil then
                do return end
              end

              local ip = string.sub(authority, 0, s-1)
              local port = string.sub(authority, s)
              local dashes, num = string.gsub(ip, "%.", "-")
              local svc = "service-per-pod-"..dashes..".cortex.svc.cluster.local"..port
              request_handle:logInfo("new authority: "..svc)
              request_handle:headers():replace(":authority", svc)
            end
  workloadSelector:
    labels:
      ip-to-service-rewrite-enabled: "true"

[dsabsay]

More details on generating the DNS entries needed for the EnvoyFilter I shared:

I created a DecoratorController with metacontroller that applies to all Pods with a pre-defined label. The DecoratorController has two hooks:

• sync - Extracts the Pod IP (available in the hook request from metacontroller). Adds the Pod IP to a label on the Pod. Then attaches a new Service object. The Service selector references the Pod IP label we just created. The name of the Service of course needs to match what the EnvoyFilter does. For the ports, my implementation extracts them from an existing label (set manually). That could probably be hard-coded. You need 9095 and (I think) 80 (for some components).
• finalize - remove all attached Services when the Pod is deleted

FWIW, my experience with metacontroller has been good. I was initially nervous about response time and general stability, but I've never had issues. On top of that, the metacontroller project receives frequent patches. I've never had problems with the upgrades.