[UX] Checkmark

[PetrochukM]

Hi There!

I am onboarding new employees onto the comet. Both of my new employees were confused by the green checkmark. They thought that an experiment was "done"; however, the green checkmark simply means that an experiment is "paused" or "stopped". It can still be restarted.

I think it'd be more clear to have a paused icon rather than a checkmark. Or some sort of icon that indicates that the current process has been killed.

Thanks!

[dsblank]

Hi @PetrochukM ! Hmmm... usually the green checkmark does mean that it is done. In what situation is the code "paused" where it could be "restarted"?

[PetrochukM]

Hi @dsblank!

We use preemptible Google Cloud instances; therefore, the machines die every 24 hours. They need to be restarted every 24 hours from a checkpoint.

The experiments are "done" when they have been training for more than 72 hours typically.

Thanks!

[dsblank]

Interesting... is there a status your code could send via the Experiment to let it know that, even though it won't be getting a regular status signal (a heartbeat that we send regularly) that the experiment will resume in the future? We could could probably accommodate that.

[PetrochukM]

Potentially! The experiment basically just crashes instead of closing gracefully. The machine is shutdown.

Maybe, Comet could detect a graceful shutdown vs a crash?

The reason we use preemptible Google Cloud instances is that it gives a 50% discount on GPU costs.

[dsblank]

We could probably show a different icon instead of a checkmark in that situation. I'll check it out...

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions[bot]]

This issue was closed because it has been stalled for 5 days with no activity.