sd-scripts icon indicating copy to clipboard operation
sd-scripts copied to clipboard

weighted captions for SDXL

Open clementine-of-whitewind opened this issue 7 months ago • 0 comments

Hi, I really, really need weighted captions and I see this in sdxl_train_network:

                # Get the text embedding for conditioning
                # TODO support weighted captions
                # if args.weighted_captions:
                #     encoder_hidden_states = get_weighted_text_embeddings(
                #         tokenizer,
                #         text_encoder,
                #         batch["captions"],
                #         accelerator.device,
                #         args.max_token_length // 75 if args.max_token_length else 1,
                #         clip_skip=args.clip_skip,
                #     )
                # else:

I've looked a little around in the scripts, looked at train_util.get_hidden_states_sdxl, sdxl_gen_img and library/custom_train_functions but I don't have any experience in this so I don't know how to implement it myself, does anyone have any suggestions how to get this working quick and dirty? there are a couple of different functions with similiar functionality already implemented but I know too little to meaningfully copy-paste them...

clementine-of-whitewind avatar Jul 23 '24 23:07 clementine-of-whitewind