ai-toolkit
ai-toolkit copied to clipboard
how to save the loss during lora finetune
This is for bugs only
Did you already ask in the discord?
Yes/No
You verified that this is a bug and not a feature request or question by asking in the discord?
Yes/No
Describe the bug
The loss_dict is in the run method of class BaseSDTrainProcess in the file jobs/process/BaseSDTrainProcess.py:
# flush()
### HOOK ###
loss_dict = self.hook_train_loop(batch_list)
You can declare a global list and record the loss history:
...
loss_history = [] # global variable
...
# flush()
### HOOK ###
loss_dict = self.hook_train_loop(batch_list)
loss_history.append(loss_dict['loss'])
...
###################################################################
## END TRAIN LOOP
###################################################################
plt.figure(figsize=(8, 4))
plt.plot(loss_history, marker='o', linestyle='-', label='Data')
plt.title('Line Chart of Float List')
plt.xlabel('step')
plt.ylabel('loss')
plt.grid(True)
plt.legend()
plt.savefig('loss.png')
self.progress_bar.close()
...