zilliztech
Want to run specific milvus pods as daemons in Kubernetes cluster
Hi, I want to run data, query and index nodes as daemons in my cluster. I am unable to figure out what can we modify from the milvus operator code to make this happen. Any one who has tried this can help me out.
My use case is I want to make these pods uniform across my nodes (each node has same number of distinct pods) and also make sure when we add new node to the cluster , we are able to scale the replica count by 1 and schedule the new pods on the new node. This made me inclined to the Daemonset approach. If anyone has more better ideas than this please suggest me.
Hi @rattipavan According to your description, one simple way I can think of is to use multiple standalone pods to form a cluster. The "standalone" here is actually the "all in one" mode. Every time you add a standalone, the computing resources of all components (proxy, datanode, indexnode, querynode) will increase. Then you need to set podAntiaffinity for the standalone to make each machine run only one pod. The only drawback is that when the k8s node increases, you need to manually add a replica of the standalone.
Two Questions here:
-
for the daemon set: If I need to use daemonset in worst case, do milvus operator support the making specific pods ((proxy, datanode, indexnode, querynode) as DaemonSets , if yes how?
-
Regarding your reply, so you meant we can add standalone pod(which has all these pods in it) which makes it easy for distribution. Will have to read on how to do this as a standalone.
- No
- There's no doc about it yet. You can specify mode as cluster, while set other components replica to 0 in Milvus CR
@haorenfsa , thanks for the reply. regarding your statement above that "The only drawback is that when the k8s node increases, you need to manually add a replica of the standalone" so as of now for milvus, there is no way for us to dynamically increase the replica count of any data or query pod is it?? only way is to do it manually .
@rattipavan Or I think you can do this. In fact, the daemonset deployment mode does not require too much operation. You can first deploy a cluster mode standalone, and then translate its deployment into DaemonSet. Later, when you officially deploy, you need to deploy a Milvus CR with replica=0 so that milvus-operator can manage your etcd, MQ, MinIO. Then deploy a separate DaemonSet that was translated above.
@haorenfsa ok sure, I will look into your approach definitely . I have question regarding the behaviour of scaling the milvus as specified in https://milvus.io/docs/scaleout.md , here it states "Currently, Milvus 2.1.0 only supports scaling in and out manually" is it true with the latest versions 2.5?
For milvus-operator, you could set replicas to -1 which indicates the operator to hand-off control of replicas. Then you can use typical HPA ways to auto scale it according to metrics.
See:
- https://github.com/zilliztech/milvus-operator/blob/main/config/samples/hpa.yaml
- https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/