wandb
Is it possible to tell wandb show my images as soon as possible?
Hi, I understand that wandb will not immediately sync and show the logged images(or other media data) for efficiency concern.
But there are situations that we did not concern the efficiency, we only want to see the logged images as soon as possible, for example, when you are debugging the model and what to see the logged images as soon as possible, maybe you will stop the run when see the images and find this run is useless instead of wait for while and stop it.
So I think it will be good to allow users to configure wandb and tell it show the images as soon as possible.
How long of a delay are you seeing? It generally shouldn't take longer than 30 seconds for any logged images to appear.
How long of a delay are you seeing? It generally shouldn't take longer than 30 seconds for any logged images to appear.
It usually takes about 30 seconds or so, like you said, and some times(not very often) takes much longer may be a few minutes, especially when I kill the run too often.
For normal run 30 seconds is not a problem, what I suggest is mainly for debug purpose, in that case we may need to run a lot of very short runs, and 30 seconds is a little a annoy :) Any way, it seems to have this customization is compete to some sense and may cover some edge use cases.
Hi @wztdream! Could adding os.system('wandb sync') to your code be useful?
Hi @wztdream! Could adding
os.system('wandb sync')to your code be useful?
Thank you, do I need to add this line for each wandb.log or just add it at the top of my code?
If you'd like to synchronize just after each training epoch, you should add it for every wandb.log()