Hyperparameter tracking

[ccharest93]

Sensitive parameters are not really DRY since they are repeated in verify_command_line_training_args, but for now only two repeats is not too bad.

The setting of hyperparameters through config is handled by Accelerate differently for each tracker so the changes are not breaking.

Tested on Wandb and tensorboard, they both work. More testing on tensorboard is welcomed as i mostly use wandb.

Related PR #792 #959 @rockerBOO #1231

@kohya-ss Lets push this through hyperparameter tracking is very useful when designing large scale experiments.

[gshawn3]

Looks great. Thank you for implementing this! I was having to take care of it through an additional script.

[gshawn3]

@rafstahelin yep, the hyperparameters will be passed as wandb config parameters for the training run and can then be used in wandb to filter, compare runs, etc. IMHO the current wandb integration is not very useful without tracking the run hyperparameters, which is why this PR is sorely needed. Hope this helps.

[rafstahelin]

@rafstahelin yep, the hyperparameters will be passed as wandb config parameters for the training run and can then be used in wandb to filter, compare runs, etc. IMHO the current wandb integration is not very useful without tracking the run hyperparameters, which is why this PR is sorely needed. Hope this helps.

yup can confirm it works. Until a merge, if there is an update, what's the best way to maintain this code when i git pull kohya_ss? Do i have to keep these bits of code to repurpose if updating the scripts is necessary? Sorry, I am by no means very ml literate

[gshawn3]

Until this PR gets merged by @kohya-ss, when there are changes in the main branch you would just need to merge those changes back into your local branch.

Might be easier to understand if I just list the git commands:

# initially, check out this PR into a local branch
git fetch origin pull/1285/head:hyperparams
git checkout hyperparams

# whenever kohya pushes updates to the main branch, merge those changes back into your branch
git switch main
git pull
git switch hyperparams
git merge main

That should do it!

[rafstahelin]

@gshawn3 thanks so much, much appreciated. This will help me

[rafstahelin]

@gshawn3 having trouble making this work within my kohya_ss install Should this work, if I am doing it inside kohya_ss install? sd-scripts is in kohya_ss. Can i run the git commands on the sd-scripts path? I tested but wasnt getting consistent updates to the code on the 3 main py scripts. Wondering if this is a tenuous approach to checking out a repo that is within a wrapper?

[gshawn3]

@rafstahelin it sounds like your issue might require some more specific assistance. I’m happy to help out, but could you please open a separate Discussion in bmaltais' repo with details, so we can follow up there? This way we can avoid derailing the PR conversation and keep this thread focused on reviewing the proposed changes.

In the meantime, it looks like this should help: https://github.com/bmaltais/kohya_ss/blob/master/examples/pull%20kohya_ss%20sd-scripts%20updates%20in.md

[rafstahelin]

@gshawn3 thanks for the helpful link. I'll give this a try. And need be will open a thread at kohya_ss.. thanks again

[rafstahelin]

@rafstahelin it sounds like your issue might require some more specific assistance. I’m happy to help out, but could you please open a separate Discussion in bmaltais' repo with details, so we can follow up there? This way we can avoid derailing the PR conversation and keep this thread focused on reviewing the proposed changes.

In the meantime, it looks like this should help: https://github.com/bmaltais/kohya_ss/blob/master/examples/pull%20kohya_ss%20sd-scripts%20updates%20in.md

hi @gshawn3 I tried your instructions. Please see https://github.com/bmaltais/kohya_ss/issues/2482

thanks

[kohya-ss]

Thank you for this PR and sorry for the delay.

I would like to confirm one thing, am I correct in understanding that this PR is to replace #1231 (we don't need to merge #1231 if we merge this).

[ccharest93]

No worries and thanks for all the work you do on sd-scripts! Yes this replaces each of the PR in my initial comment, #959 was much better written than #1231 so i pulled more heavily on that code and implement a sensitive information filter as per your request in #1231.

[kohya-ss]

Thank you for clarification! I will merge this sooner.!

[kohya-ss]

Sorry for the delay. I added --log_config option to enable this feature. I appreciate your understanding.