milvus
milvus copied to clipboard
[Bug]: Standalone oomkilled after upgrading mater branch image
Is there an existing issue for this?
- [X] I have searched the existing issues
Environment
- Milvus version: e83ecd50 -> master-20240621-4e414fb7-amd64
- Deployment mode(standalone or cluster): standalone
- MQ type(rocksmq, pulsar or kafka): rocksmq
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS):
- CPU/Memory:
- GPU:
- Others:
Current Behavior
- standalone pod
zong-go-master-milvus-standalone-5cc75d965b-t46jrwith resource config
Limits:
cpu: 8
memory: 16Gi
Requests:
cpu: 5
memory: 9Gi
- upgrade image to
master-20240621-4e414fb7-amd64and pod oomkilled - upgrade standalone pod memory limit to 32Gi and pod is running
- pod memory usage
Expected Behavior
No response
Steps To Reproduce
No response
Milvus Log
pods:
zong-go-master-etcd-0 1/1 Running 0 3d 10.104.25.21 4am-node30 <none> <none>
zong-go-master-milvus-standalone-6cdbf578d-phtls 1/1 Running 0 10m 10.104.19.7 4am-node28 <none> <none>
zong-go-master-minio-55956b966c-s48rt 1/1 Running 0 3d 10.104.34.72 4am-node37 <none> <none>
Anything else?
No response
there are only 20 collections with totally 15k vectors loaded before upgrading, btw
pyroscope of pod zong-go-master-milvus-standalone-6cdbf578d-phtls: http://10.100.36.158:4040/?query=zong-go-master-milvus-standalone.alloc_objects%7B%7D&from=1718955620&until=1718959425
So we have 20collections with only 15K vectors each collection and 16GB standalone get OOM?