Docs: Add compatibility information about running Mimir with Istio

[09jvilla]

Create a small section of the docs that mentions if Mimir is compatible with Istio and if so, what are the considerations/limitations/requirements.

We've had at least 1 OSS user ask about using Istio sidecars to secure traffic between the Mimir components.

And there were a few asks around Istio support for Cortex, the parent project to Mimir.

Not sure where this would best live, but @osg-grafana can perhaps weigh in.

[batistein]

I'm also very interested. Especially how you could use istio for auth.

[pstibrany]

I'm also very interested. Especially how you could use istio for auth.

I am afraid none of Mimir maintainers have experience with running Mimir with Istio at the moment.

[agalue]

Based on my experience, having Mimir on top of Istio won't be straightforward because Istio doesn't support Pod-to-Pod communication. It expects that all traffic goes between Kubernetes services (or Service-to-Service). However, Linkerd doesn't have that limitation (it does support Pod-to-Pod communication), so we opted for using it instead of Istio.

Several components of Mimir need Pod-to-Pod communication; for instance, distributors need to talk to specific ingesters (using their Pod IP) and cannot use the "ingester service".

Here is a PoC I created showing how the LGTM stack works with Linkerd: https://github.com/agalue/LGTM-PoC

[osg-grafana]

Closing because the action item(s) for documentation are unclear.