detecto icon indicating copy to clipboard operation
detecto copied to clipboard

Improve the progress bar during training (i did a improvement)

Open Fatichti opened this issue 2 years ago • 0 comments

Describe the feature you'd like Training is the most time-consuming part of any reconnaissance script. The progress bar currently set, only shows the time remaining per epoch, why not show the estimated total time with the estimated end time?

Describe the use cases of the feature This would be very useful to know if the pc needs to run more at night, etc... Typically, you can (what I did) do a complete chain of processing (training + analysis on images + results on Excel) all automatically and therefore having the time (or date) of completion of this process.

This is very useful and improves efficiency, you know when the work is finished, you can do something else on the side

Additional context I have applied an improvement to make these time estimates, thanks to this, you can :

  • get the total time of the training
  • get the end time (you can put a date instead if you want)

🤜 Here is the step to reproduce :

Open the core file in the detecto folder :

cd $PATH_TO_DETECTO
code core.py  // because i have visual studio code installed

Go to line 500, you should see :

# Train on the entire dataset for the specified number of times (epochs)
for epoch in range(epochs):
    if verbose:
        print('Epoch {} of {}...

Edit as follow :

self.totalTime = 0
for epoch in range(epochs):
    timeEpoch = 0
    if self.totalTimel == 0:
        TotalTimeFormat = "Calculation...."
        EndHourFormat = "Calculation... (available at epoch 2)"
    else:
        now = time.time()
        EndHour = now + self.totalTime
        EndHourFormat = time.strftime("%H:%M:%S",time.localtime(EndHour))
        TotalTimeFormat = time.strftime("%H:%M:%S",time.gmtime(self.totalTime))
    if verbose:
        print('Epoch {} of {} - Total time estimated :{} - End hour estimated :{}'.format(epoch + 1, epochs, TotalTimeFormat, EndHourFormat))

Then, a few lines down, you should see :

# Validation step
if val_dataset is not None:
  avg_loss = 0
  ...

Add before these lines this :

timeEpoch = iterable.format_dict['elapsed']

After, in the same idea also for the validation part, around line 580, you should see :

# Update the learning rate every few epochs
lr_scheduler.step()

Add after this line this :

timeEpoch = timeEpoch + iterable.format_dict['elapsed']
self.totalTime = timeEpoch * (epochs - epoch)

🎉 It's done !

OUTPUT (it's french because i am french sory but you have an idea of the output) image

Note : As you can see, i add anothers things to get an output more readable :

  • print("\n") (just after the for epoch in range(epochs))
  • i have replaced the print('Loss: {} ' format...) by print('Loss: %.4f' % avg_loss)

Fatichti avatar Jun 15 '22 10:06 Fatichti