spark-operator
spark-operator copied to clipboard
How many jobs can spark operator process at a time?
I have some jobs running on spark operator but some of the jobs staying in "PENDING_RERUN" for several days, Is there any limit on how many jobs spark operator can process at a time?
Another thing i am seeing as a lot of executors are staying in UNKNOWN status "da31-exec-5872": "UNKNOWN", da31-exec-5873": "UNKNOWN", "da31-exec-5874": "UNKNOWN", "da31-exec-5875": "UNKNOWN", "da31-exec-5876": "UNKNOWN", "da31-exec-5877": "UNKNOWN", "da31-exec-5878": "UNKNOWN",
All i can see in the logs is millions of executors from couple of jobs in UNKNOWN status and some of them in COMPETED, could that be a part of problem?