Han Liu

Do you guys find any solution?

I met the similar problems, and after I pass --precision 16-mixed, the problem still exist @carmocca

@shuwang127 The first loss is nan, and it soon become OOM

I still get a OOM using current main, here is my setting @carmocca : Loading model 'checkpoints/tiiuae/falcon-7b/lit_model.pth' with {'block_size': 1024, 'vocab_size': 50254, 'padding_multiple': 512, 'padded_vocab_size': 65024, 'n_layer': 32, 'n_head': 71,...

> Hi @carmocca I've tried with the recommended changes it works fine for `adapter.py` for first epoch but after first epoch get's completed the loss automatically changes to NAN. >...

I also got the bad qulity results, have you solved it yet?

The problems seems lie in the function: class CausalConv1d(nn.Conv1d): def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) self.causal_padding = self.dilation[0] * (self.kernel_size[0] - 1) def forward(self, x): return self._conv_forward(F.pad(x, [self.causal_padding, 0]), self.weight,...

First train the discriminator then train the generator will solve this problem, but I am sure whether this will affect the training accuracy. I have read many GAN implementations, and...

> > Can you get a good result? I haven't started training yet, but I will update my results here as soon as I did it.

> Hi, Thanks for your issue! I encountered this issue with inplace operation on other parts of the code. Some other might need a fix for that. Regarding the training...