[Bug]: Does not remove embeddings from memory when updating

[mykeehu]

Is there an existing issue for this?

• [X] I have searched the existing issues and checked the recent builds/commits

What happened?

Embeddinget edzek. If I reload a previous move and then press Refresh, the embeddings are not removed before reloading, so they accumulate in memory each time I refresh, affecting the workout. I get completely different results for the same parameters if I restart the SD or just refresh a few times. Therefore, I suspect that when updating, it should first be unloaded from memory and then reloaded again.

Steps to reproduce the problem

1. create an embedding with e.g. 7 vectors
2. start training subject to 500 with these LR parameters: 5e-04:200, 5e-05:500, 5e-06:800, 5e-07:1000
3. stop at 500 and generate an image at txt2img.
4. delete the embedding and go back to an earlier step, e.g. from 300.
5. on the Train tab, press an update!
6. train again up to 500, then generate the same image!
7. repeat the step back with other points (e.g. from 400 or 500) and you will see the difference in the txt2img tab.
8. restart the SD and regenerate the embedding and train again up to 500. It should resemble the first version.

What should have happened?

Distorted the TI training when it generated black and white images 4x in a row based on color images and lost color, suspected that updates were causing the problem. After rebooting SD, creating a new workout "fixed" it.

Commit where the problem happens

f894dd552f68bea27476f1f360ab8e79f3a65b4f

What platforms do you use to access UI ?

Windows

What browsers do you use to access the UI ?

Google Chrome

Command Line Arguments

--xformers --ui-config-file 1my/ui-config-my.json --ui-settings-file 1my/config-my.json --autolaunch

Additional information, context and logs

No response

[denliner]

That vram leak issue does not only concern embeddings it also happen during hypernetwork training.

[mykeehu]

In the latest version (1ed227b3b57f06f4152be3bfc9f83b0a839a2604) the problem with embedding was so bad that it was generating noise on the third model, nothing to do with the images. After closing SD command I deleted the model, restarted SD, generated the model again and got usable results.

It could also be related to the fact that when I do [name], [filewords] training, it only takes the text after the comma, [name] does not return anything from the training in txt2img, it generates unrealistic images, like it is full of garbage

[mykeehu]

When I stop the training and then create the file again using the "Overwrite Old Embedding" option, the embedding is not reloaded, the training does not start from the beginning, but continues from the previous step. I would need to revisit the embedding function because it is getting more and more messy and cross-training.

[catboxanon]

Closing as stale.