Progress Bar for Jobs (Implementation)

[yuvalkirstain]

Hi :)

Thanks for creating this awesome open source repo, it helps me a lot!

I wrote a function that tracks jobs` status with a progress bar. Perhaps it will save users a bit of time to implement it by themselves :)

def track_jobs_with_pbar(jobs):
    num_completed = 0
    with tqdm(total=len(jobs)) as pbar:
        while any(job.state not in ["COMPLETED", "FAILED", "DONE"] for job in jobs):
            time.sleep(2)
            state2count = Counter([j.state for j in jobs])
            newly_completed = state2count["COMPLETED"] - num_completed
            pbar.update(newly_completed)
            num_completed = state2count["COMPLETED"]
            s = [f"{k}: {v}" for k, v in state2count.items()]
            pbar.set_description(" | ".join(s))

It looks like this:

[gwenzek]

thanks for sharing, this is better than what we have. I think we could try to make the API of "as_completed" nicer. Another PR with custom job reporting just went through: #1675

I'd like to take a step back and think about a simple api that could provide reasonable defaults and also allow custom extensions.

Maybe def as_completed(jobs, tqdm=False, on_job_done=None)

[yuvalkirstain]

@gwenzek how would you like me to help and add this option to submitit?

[FabricioArendTorres]

Just as a comment:

there are other states than ["COMPLETED", "FAILED", "DONE"] which would indicate that a job is neither pending nor running. At the very least I also encountered OUT_OF_MEMORY", which results in the function above being in an endless loop. So for fixing this script just add it to the list.

[gwenzek]

Thanks for the suggestion but, for now, I think this kind of helpers is better as per project utils. Not everyone want to use tqdm.