guaneec

Had to change the shuffle function as well because somehow indexing numpy arrays with torch tensors of length 1 is a special case.

Remove activation from final layer of Hypernetworks

7

From what I understand, a Hypernetwork learns how to nudge the context in Cross Attention mechanisms. ```python # modules/hypernetworks/hypernetwork.py#L89 def forward(self, x): return x + self.linear(x) * self.multiplier ``` I...

Custom Diffusion implementation

4

A very rudimentary Custom Diffusion implementation ## What is Custom Diffusion [Custom Diffusion](https://www.cs.cmu.edu/~custom-diffusion/) is, in short, finetuning-lite with TI. Instead of tuning the whole model, only the K and V...

Alternative batching behavior for mix-sized training

8

Continuing #6620 In this version, the dataset is batched almost like there is no bucketing. Internally, a batch is replaced with a superbatch consisting of one or more batches. I...