Superset helm chart not following the resources setup

[rnpaiva]

A clear and concise description of what the bug is.

How to reproduce the bug

Change the resources limits for the worker and deploy superset 3.0.2 in eks

Expected results

Resource limit as setup .

resources: 
    limits:
     cpu: 1
     memory: 3Gi
    requests:
     cpu: 1
     memory: 2Gi

what you expected to happen.

Actual results

The workers are requesting a lot of memory and cpu that it's not setup . Superset it's running and I can open and create whatever I need , but the last worker it's having OOM .

Screenshots

( attached )

[rnpaiva]

I'm not sure what is happening here ! All my helm chart it's setup with resources and limits , but when I deploy to kubernetes the limit and the request it's way high than what I setup

[shakeelansari63]

I think the issue occurs because celery spawns multiple processes which kubernetes does not like some how. And it starts killing the celery worker pods.
I have also noticed this issue in AKS. The fix I found was to use --pool solo after the worker command.

This Gist mentions same issue and the fix - https://gist.github.com/chris-w-jarvis/b96e21edc8a1a7c076797b3e95786850

[shakeelansari63]

I went with Pool Solo, but it seems --pool gevent is much more performant. But I haven't tried it.

[shakeelansari63]

I am happy to submit PR if you thing the fix --pool gevent makes sense.

[rusackas]

CC @villebro here regarding the proposed PR, and whether or not this might still be an issue worth keeping open (vs closing as stale, since it's been ~4 months)

[shakeelansari63]

I had also started a discussion, but it did not get any response.

https://github.com/apache/superset/discussions/27070