spark-operator icon indicating copy to clipboard operation
spark-operator copied to clipboard

How many jobs can spark operator process at a time?

Open roligupt opened this issue 1 year ago • 0 comments

I have some jobs running on spark operator but some of the jobs staying in "PENDING_RERUN" for several days, Is there any limit on how many jobs spark operator can process at a time?

Another thing i am seeing as a lot of executors are staying in UNKNOWN status "da31-exec-5872": "UNKNOWN", da31-exec-5873": "UNKNOWN", "da31-exec-5874": "UNKNOWN", "da31-exec-5875": "UNKNOWN", "da31-exec-5876": "UNKNOWN", "da31-exec-5877": "UNKNOWN", "da31-exec-5878": "UNKNOWN",

All i can see in the logs is millions of executors from couple of jobs in UNKNOWN status and some of them in COMPETED, could that be a part of problem?

roligupt avatar Feb 05 '24 22:02 roligupt