pangeo-cloud-federation icon indicating copy to clipboard operation
pangeo-cloud-federation copied to clipboard

Published 20 hours ago • verified publisher icon pangeo-data

how much cpu / memory can notebook pods use?

Open rabernat opened this issue 5 years ago • 0 comments

The biggest possible pod we allow in ocean.pangeo.io is defined by the profile_list entry:

https://github.com/pangeo-data/pangeo-cloud-federation/blob/cfe82751b7e52ab665b4b4a4bd2278b05776d253/deployments/ocean/config/common.yaml#L53-L58

We have a nodepool with n1-highmem-16 (16 vCPUs, 104 GB memory) nodes. However, when I try to launch the x-large profile, the event log shows

Server requested
2019-03-06 17:49:34+00:00 [Warning] 0/3 nodes are available: 2 Insufficient cpu, 3 Insufficient memory.
2019-03-06 17:49:48+00:00 [Normal] pod didn't trigger scale-up (it wouldn't fit if a new node is added)

How much headroom do we need between the pod resource requests and the node capacity? I would think that 14 cpus and 96GB of memory would fit on a 16 vCPU / 104GB memory node. How can we debug this?

rabernat avatar Mar 06 '19 17:03 rabernat