root_painter icon indicating copy to clipboard operation
root_painter copied to clipboard

users should be warned when they try to start training when a model is already training

Open Abe404 opened this issue 1 year ago • 4 comments

right now if a model is training and the user clicks 'start training' for a different project, training does not start (for the new project) as training is already running, but the user is not told the reason why training does not start.

They should at least be made aware that training was not started because 'training already running for project example_project' or similar.

Abe404 avatar Apr 04 '24 14:04 Abe404

I’ll add some code to print a warning on the terminal in the trainer

Felipegalind0 avatar May 28 '24 08:05 Felipegalind0

That's a good idea, but in some cases users are running RootPainter purely from the client and only a sysadmin has access to the trainer.

There's a mechanism to send messages to the client that could be used. I'm also wondering if we could add something a little more obvious i.e some type of alert box that pops up in the client telling them that training is already running (the current messages can sometimes be missed, as they get overwritten by the next message).

Just to clarify: The issue is for users that start training for a project where training is already running on the server for a different project. This is an issue that comes up when multiple users are sharing a server.

I'm not 100% sure what the best solution for this is. Basically if users start training (from the client) when the server is already running training for a different user the client should at least let them know what the problem is somehow (in the client, as they might not see the server output).

Abe404 avatar May 28 '24 09:05 Abe404

This could also happen in a single user case.

For example, if a user follows the following steps:

  1. User starts training for their project.
  2. User opens a new project
  3. User starts training in the new project (Training won't work because trainer is busy with the previous project but the user doesn't get any feedback in the client).

Abe404 avatar May 28 '24 09:05 Abe404

Another option is we rethink how to give the user info about the server status via the client i.e have a custom module (GUI widget) that allows the user to stop/start the server, view server messages, get notifications/warnings/errors from the server etc without looking at the command line. This is a bit more involved and would require more discussion/design.

Abe404 avatar May 28 '24 10:05 Abe404