tensorboardX icon indicating copy to clipboard operation
tensorboardX copied to clipboard

How can I connect previous log file into new one in tensorboard

Open mohanades opened this issue 2 years ago • 1 comments

Hi, I trained my model for 50 epochs. Now I want to continue training for other 15 epochs. so I start training with model.load_from_checkpoint("path") but I don't know how to show continuation of logs with tensorboard I thought if I write same path of previous file logs, it continue but it didn't show me the previous logs.

I'd be appreciate if you could help me because I need to see the previous training logs along with the new ones. Part of source code is:

checkpoint_callback=ModelCheckpoint(dirpath="model",filename="newmodel_1",save_last=True,verbose=True,monitor="val_ce_loss",mode="min")
logger=TensorBoardLogger("training-logs",name="test_1")
model1 = model(config)
model_with_previous = model1.load_from_checkpoint(base_path_ckpt)
trainer = pl.Trainer(
     max_steps = max_steps,
     max_epochs=N_EPOCHS,
     gpus=(1 if torch.cuda.is_available() else 0),
     logger=logger,
     callbacks=[checkpoint_callback], 
)
%load_ext tensorboard
%tensorboard --logdir ./lightning_logs

mohanades avatar Dec 31 '22 03:12 mohanades

Hi, you might need to pass the purge_step parameter. Here is an low level API example: https://github.com/lanpa/tensorboardX/blob/master/examples/demo_purge.py

lanpa avatar Jan 02 '23 06:01 lanpa