noobaa-core
noobaa-core copied to clipboard
noobaa
Without IO, endpoint consumes 800MB throughout
Environment info
- NooBaa Version: master-20210728
- Platform: OCP 4.6.16
Actual behavior
- Idle system Grafana view showed 800+MB consumption by endpoint pod. CPU usage is 1% which is OK
Expected behavior
- Endpoint pod should consume much less memory as it is 40% of the overall memory.
Steps to reproduce
- Captured Grafana view and looked at top command from endpoint pod
Copying Grafana o/p and must-gather o/p along with noobaa diagnose
More information - Screenshots / Logs / Other output
@akmithal How much memory would you expect the endpoint pod to consume? You can define endpoint resource limit in the NooBaa CR (spec.endpoints.resources... - doc) Or you mean that you expect the endpoints to shrink their heap size more aggressively after I/O calms down?
Hello @guymguym , I found 40% consumption of 2 GB limit in idle environment to be very high. My reference point is, on new installation, it stayed around 200-300MB. On doing IO, it peaks upto 1.5 GB's but I was expecting it to come down to initial memory limit post installation. I didn't change any resource value in the configuration.
I suspect such consumption in idle setup would result in huge scaling of endpoints resulting in proportionate allocation of memory.
Does this consumption of resources look OK to you ?
@jeniawhite had a thought regarding our 500MB buffers for NS FS which might be affecting this as well. Those are not dynamic but flat 500...
Hi @nimrod-becker , I am finding that Memory resources were not freed up after IO was stopped. I have 3 GB RAM defined in the configuration. During IO, usage went upto 1.8GB and it didn't drop after that.
@romayalon can we have the JIRA ticket mentioned if this going to be implemented
@nimrod-becker , could you screen and assign
There were multiple fixes to memory handling in the endpoint by Guy and Jacky, can we please rerun with a new build ?