ai-toolkit icon indicating copy to clipboard operation
ai-toolkit copied to clipboard

how to save the loss during lora finetune

Open MathamPollard opened this issue 1 year ago • 1 comments

This is for bugs only

Did you already ask in the discord?

Yes/No

You verified that this is a bug and not a feature request or question by asking in the discord?

Yes/No

Describe the bug

MathamPollard avatar Nov 12 '24 01:11 MathamPollard

The loss_dict is in the run method of class BaseSDTrainProcess in the file jobs/process/BaseSDTrainProcess.py:

            # flush()
            ### HOOK ###
            loss_dict = self.hook_train_loop(batch_list)

You can declare a global list and record the loss history:

...
loss_history = []  # global variable


...
            # flush()
            ### HOOK ###
            loss_dict = self.hook_train_loop(batch_list)
            loss_history.append(loss_dict['loss'])

...
        ###################################################################
        ##  END TRAIN LOOP
        ###################################################################

        plt.figure(figsize=(8, 4))
        plt.plot(loss_history, marker='o', linestyle='-', label='Data')
        plt.title('Line Chart of Float List')
        plt.xlabel('step')
        plt.ylabel('loss')
        plt.grid(True)
        plt.legend()
        plt.savefig('loss.png')

        self.progress_bar.close()

...

Poseidon-fan avatar Nov 12 '24 07:11 Poseidon-fan