vcluster
vcluster copied to clipboard
loft-sh
Epic: Add support for logging solutions inside vcluster
Use Case
Currently its not possible to use common logging solutions such as FluentD or ELK stack inside vcluster, because all of them mount the node host path directly. While an acceptable solution is to provide a single logging solution in the host cluster, there are also use cases where you would want to install the solution directly inside the virtual cluster.
Implementation
We can rewrite the host path with the correct pod uids trough symlinks in a sidecar container. We need to continously monitor changes to that folder and reflect it in the fake host path folder.
This is something crucial for us. Do you have an ETA on when this will be implemented?
As always: great product!
It's worth noting we solved this problem (for restic (velero)) with https://github.com/getoutreach/vcluster-fs-syncer. Of course it's not perfect but worth pointing out :D
@FabianKramm I want to mention a related issue that might make us move fluentd (and all daemonsets) to the host cluster regardless.
When there are a lot of vclusters on the same host cluster, as the number of nodes increases (cluster autoscaler scale-up), there are a lot of fluentd pods and all other daemonset pods overhead on each node for each vcluster. For example, if we have 10 vclusters, 5 daemonsets for each vcluster, and 15 nodes, then each node has 10 * 5 pods just for the daemonsets, and for each vcluster, there are 5 * 15 daemonset pods to keep in sync.
These pods themselves use a lot of compute resources on each node, but I guess that's not a problem due to vcluster. But this has caused OOM on the vcluster management pod ([vcluster-name]-0, I'm not sure how you call this pod. It's the one that has vcluster and syncer containers). Take the above example, each vcluster needs to manage 75 daemonset pods + all other pods. We have increased the memory limit of the syncer pod to 5G and it starts to fail when there are 8+ vclusters.
So curious on how the syncer manages all its pods, especially in terms of memory consumption. Have you done any load/scale test?
If you think this is a separate issue, I can create another issue just for it. Thank you
@ddl-risong-na yes that is expected. One way to work around this is to use a node selector inside the vcluster that only syncs nodes that fullfill the selector to the virtual cluster (see the docs for more info). If you really want to sync all nodes to the vcluster, you need to make sure that DaemonSets mostly run on the host cluster as this is a lot more efficient as you already have discovered.
Regarding the vcluster pod failing, that is probably a different issue, but you are correct it usually should withstand quite a lot of load. We did some improvements for this in the latest version v0.9.1, so if you are not using that yet would be interesting to know if that also goes into an OOM state. If yes, this is definitely worth creating a new issue for, as vcluster should be able to scale to several hundreds if not thousands per cluster.
I installed Elasticsearch on the host and what I got from search is following and it shows nothing related to kubernetes labels/namespace
What is a workaround for this?
@FabianKramm are you looking at vcluster plug-in to solve this? Can I take a stab at it? If you can give more details it would be great
