chore: adding training parameters to wandb ai experiments

[plucked]

This change adds the parameters used with accelerate and sends them as a payload to wandb.ai. That allows the parameters to show up in the table of runs and helps to get a better understanding of what the difference is between runs. It also uses the model's name as run name to give more context.

[saunderez]

You are a champion, I was looking at the wandb docs on the weekend because I wanted exactly this but got sidetracked when and never got back to it. Will pull and confirm everything works.

[rafstahelin]

Is there a way to set the Project Name in Additional Parameters of Kohya_ss rather than in train_utils?

[kohya-ss]

Thank you for this! I was checking the code before the merge and found the following code in accelerate.

https://github.com/huggingface/accelerate/blob/a87c95da9e3b416fb10a0e7dac7d397c015c3ed5/src/accelerate/accelerator.py#L2373

It is seemed to be better to use init_kwargs argument in init_trackers for logging configs to wandb.

Is it possible to rewrite this PR to use it? I think I can implement it, but unfortunately I'm not using wandb personally, I cannot test it.

[rafstahelin]

is this going to be merged?

[rockerBOO]

Kohya is suggesting to move the config from wandb to the accelerator interface via init_kwargs/config as starting the wandb init outside the accelerator hook could cause side effects.

accelerator.init_trackers(
         project_name=args.output_name, 
         config=args
         init_kwargs={},
)

Then we could add a new argument to pass additional variables to wandb.

accelerator.init_trackers(
         project_name=args.output_name, 
         config=args,
         init_kwargs={'wandb': args.wandb_init_args},
)

Which would allow us to pass notes, tags and more described wandb.init

[deepdelirious]

accelerator.init_trackers has the config argument just for this purpose. The wandb implementation calls wandb.config.update

[rockerBOO]

I went through where we were using init_tracker already and added the args there. See #959

[kmacmcfarlane]

I also recently looked into how to do this. Great to see there's a PR already in-flight!

[FurkanGozukara]

Are you guys able to get any meaningful stats from logs for overtraining, undertraining?

[rafstahelin]

Can anyone offer me guidance?

I usually used

if args.wandb_api_key is not None: wandb.login(key=args.wandb_api_key) wandb.init(project="t1k0", name=args.output_name, config=args)

But now I am no longer getting the project name customised in wandb.

Has something changed in the code? Anyone?

[rafstahelin]

Kohya is suggesting to move the config from wandb to the accelerator interface via init_kwargs/config as starting the wandb init outside the accelerator hook could cause side effects.

accelerator.init_trackers(
         project_name=args.output_name, 
         config=args
         init_kwargs={},
)

Then we could add a new argument to pass additional variables to wandb.

accelerator.init_trackers(
         project_name=args.output_name, 
         config=args,
         init_kwargs={'wandb': args.wandb_init_args},
)

Which would allow us to pass notes, tags and more described wandb.init

how do i adjust the code now if there has been changes?

[rafstahelin]

Kohya is suggesting to move the config from wandb to the accelerator interface via init_kwargs/config as starting the wandb init outside the accelerator hook could cause side effects.

accelerator.init_trackers(
         project_name=args.output_name, 
         config=args
         init_kwargs={},
)

Then we could add a new argument to pass additional variables to wandb.

accelerator.init_trackers(
         project_name=args.output_name, 
         config=args,
         init_kwargs={'wandb': args.wandb_init_args},
)

Which would allow us to pass notes, tags and more described wandb.init

should this work in the additional parameters args: --log_tracker_name "finetuning" --wandb_run_name "myrun" ?

[VelocityRa]

Opened #1231 for this.

For now, @rafstahelin you can add the following above the accelerator.init_trackers call:

init_kwargs["wandb"] = {"name": args.output_name, "config": args}

[rafstahelin]

No longer necessary m. It sheen incorporated to gui a few releases back

[VelocityRa]

Not by default, you need to make a config for it afaict (of which I've seen no examples).

My PR makes it so if log_tracker_config isn't specified, all training options are exported out of the box.

[rafstahelin]

Sorry, it has been committed on v23.0.12 Support for Wandb project and run name has been added to the gui Best

thanks

best regards, raf

On Mon, 1 Apr 2024 at 15:26, Nick Renieris @.***> wrote:

Not by default, you need to make a config for it afaik (of which I've seen no examples).

My PR makes it so if log_tracker_config isn't specified, all training options are exported out of the box.

— Reply to this email directly, view it on GitHub https://github.com/kohya-ss/sd-scripts/pull/792#issuecomment-2029757311, or unsubscribe https://github.com/notifications/unsubscribe-auth/APNXTFVA6BHPWNQETB6KTUDY3FOAFAVCNFSM6AAAAAA4BIVWR6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMRZG42TOMZRGE . You are receiving this because you were mentioned.Message ID: @.***>

[VelocityRa]

I'm on the latest kohya_ss version, and training parameters are not sent to wandb without my changes. I don't care much about just the run name.

Also we are on the scripts (library) repo, not the GUI. Such change should happen here due to the way the wandb API works. Not the GUI.

[rafstahelin]

above the accelerator.init_trackers call:

you're absolutely right. Hadnt noticed the parameters were not being sent

[rafstahelin]

I'm on the latest kohya_ss version, and training parameters are not sent to wandb without my changes. I don't care much about just the run name.

Also we are on the scripts (library) repo, not the GUI. Such change should happen here due to the way the wandb API works. Not the GUI.

right on thank you

[rafstahelin]

Not by default, you need to make a config for it afaict (of which I've seen no examples).

My PR makes it so if log_tracker_config isn't specified, all training options are exported out of the box.

cool will be running with your suggestion for the time being. do you know when it will be merged?

[rafstahelin]

is it possible to use bothM

init_kwargs["wandb"] = {"config": args} in sdxl_train.py and WANDB run name for model name Log tracker name for project name

So as to take advantage of both?

On Mon, Apr 1, 2024 at 6:07 AM Nick Renieris @.***> wrote:

Opened #1231 https://github.com/kohya-ss/sd-scripts/pull/1231 for this.

For now, @rafstahelin https://github.com/rafstahelin you can add the following above the accelerator.init_trackers call:

init_kwargs["wandb"] = {"name": args.output_name, "config": args}

— Reply to this email directly, view it on GitHub https://github.com/kohya-ss/sd-scripts/pull/792#issuecomment-2029122474, or unsubscribe https://github.com/notifications/unsubscribe-auth/APNXTFWMBLL2GFWLZRY6DB3Y3DMQFAVCNFSM6AAAAAA4BIVWR6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMRZGEZDENBXGQ . You are receiving this because you were mentioned.Message ID: @.***>

-- thanks

best regards, raf

[rafstahelin]

in sdxd_train.py

do i remove the line: init_kwargs = toml.load(args.log_tracker_config) and add instead: init_kwargs["wandb"] = {"config": args} [image: image.png]

?

On Mon, Apr 1, 2024 at 6:07 AM Nick Renieris @.***> wrote:

Opened #1231 https://github.com/kohya-ss/sd-scripts/pull/1231 for this.

For now, @rafstahelin https://github.com/rafstahelin you can add the following above the accelerator.init_trackers call:

init_kwargs["wandb"] = {"name": args.output_name, "config": args}

— Reply to this email directly, view it on GitHub https://github.com/kohya-ss/sd-scripts/pull/792#issuecomment-2029122474, or unsubscribe https://github.com/notifications/unsubscribe-auth/APNXTFWMBLL2GFWLZRY6DB3Y3DMQFAVCNFSM6AAAAAA4BIVWR6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMRZGEZDENBXGQ . You are receiving this because you were mentioned.Message ID: @.***>

-- thanks

best regards, raf

[VelocityRa]

see my PR: https://github.com/kohya-ss/sd-scripts/pull/1231/files#diff-62cf7de156b588b9acd7af26941d7bb189368221946c8b5e63f69df5cda56f39R705-R707

then: log_tracker_name -> wandb project name wandb_run_name -> wandb run name don't specify log_tracker_config

[rafstahelin]

see my PR: https://github.com/kohya-ss/sd-scripts/pull/1231/files#diff-62cf7de156b588b9acd7af26941d7bb189368221946c8b5e63f69df5cda56f39R705-R707

then: log_tracker_name -> wandb project name wandb_run_name -> wandb run name don't specify log_tracker_config

awesome that's working now, thanksalthough, there seems to be an issue now with samplingit doesn't work on steps, only on Sample every x epochs.  Is this possible? 

[rafstahelin]

see my PR: https://github.com/kohya-ss/sd-scripts/pull/1231/files#diff-62cf7de156b588b9acd7af26941d7bb189368221946c8b5e63f69df5cda56f39R705-R707

then: log_tracker_name -> wandb project name wandb_run_name -> wandb run name don't specify log_tracker_config

awesome that's working now, though not all. And what's worst, I am not seeing samples being processed anymore. sdxl_train.zip

And not only that i dont see all the parameters

I've added my sdxl_train.py Maybe i am doing something wrong in the code i added

So for the moment, i will stick to the limited gui ouptut to wandb, unless you could possibly point out my obvious mistake

thank you

[rafstahelin]

How would I test your PR? Can I switch to your branch so I don't have to go in and modify a bunch of files manually?

Apologies for my newbie-ness

On Wed, Apr 3, 2024 at 4:17 PM Nick Renieris @.***> wrote:

see my PR: https://github.com/kohya-ss/sd-scripts/pull/1231/files#diff-62cf7de156b588b9acd7af26941d7bb189368221946c8b5e63f69df5cda56f39R705-R707

then: log_tracker_name -> wandb project name wandb_run_name -> wandb run name don't specify log_tracker_config

— Reply to this email directly, view it on GitHub https://github.com/kohya-ss/sd-scripts/pull/792#issuecomment-2034749019, or unsubscribe https://github.com/notifications/unsubscribe-auth/APNXTFS3MD634O7UHMHFRSLY3QFQTAVCNFSM6AAAAAA4BIVWR6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMZUG42DSMBRHE . You are receiving this because you were mentioned.Message ID: @.***>

-- thanks

best regards, raf