submitit
submitit copied to clipboard
Progress Bar for Jobs (Implementation)
Hi :)
Thanks for creating this awesome open source repo, it helps me a lot!
I wrote a function that tracks jobs` status with a progress bar. Perhaps it will save users a bit of time to implement it by themselves :)
def track_jobs_with_pbar(jobs):
num_completed = 0
with tqdm(total=len(jobs)) as pbar:
while any(job.state not in ["COMPLETED", "FAILED", "DONE"] for job in jobs):
time.sleep(2)
state2count = Counter([j.state for j in jobs])
newly_completed = state2count["COMPLETED"] - num_completed
pbar.update(newly_completed)
num_completed = state2count["COMPLETED"]
s = [f"{k}: {v}" for k, v in state2count.items()]
pbar.set_description(" | ".join(s))
It looks like this:
![image](https://user-images.githubusercontent.com/57996478/164989879-ee318d5e-16c8-4ea2-8c25-96e15f24eeff.png)
thanks for sharing, this is better than what we have. I think we could try to make the API of "as_completed" nicer. Another PR with custom job reporting just went through: #1675
I'd like to take a step back and think about a simple api that could provide reasonable defaults and also allow custom extensions.
Maybe def as_completed(jobs, tqdm=False, on_job_done=None)
@gwenzek how would you like me to help and add this option to submitit?
Just as a comment:
there are other states than ["COMPLETED", "FAILED", "DONE"]
which would indicate that a job is neither pending nor running.
At the very least I also encountered OUT_OF_MEMORY"
, which results in the function above being in an endless loop.
So for fixing this script just add it to the list.
Thanks for the suggestion but, for now, I think this kind of helpers is better as per project utils. Not everyone want to use tqdm.