nll is nan in training

[zhiwei-liang]

Hi, I trained the GaussianMixture model with pycave v3.2.1, however, after training a few epochs nll=nan.0 appears. And this situation happens occasionally, I would like to ask what is the solution. My training code is as follows:

representation = torch.tensor(representation) 
representation = representation.to(self.device)
gmm = GMM(num_components=3, covariance_type="diag", init_strategy="kmeans", batch_size=16, trainer_params=dict(accelerator='gpu', devices=1, max_epochs=50))
gmm.fit(representation)

The error log is as follows: Epoch 1: 100%|██████████| 15/15 [00:00<00:00, 47.29it/s, nll=-5.33e+3] Epoch 1: 100%|██████████| 15/15 [00:00<00:00, 47.23it/s, nll=-5.33e+3] Epoch 1: 100%|██████████| 15/15 [00:00<00:00, 46.75it/s, nll=nan.0]
Epoch 1: 100%|██████████| 15/15 [00:00<00:00, 46.04it/s, nll=nan.0]

Thank you so much!

Best regards, Zhiwei

[hashim19]

Hi, were you able to figure out the solution to this problem? I also trained a GMM model and nll is nan in training.

[AndreasGerken]

I have the same issue which makes the package unusable for me sadly :/ I did a optimization for n_components and just at the optimum (~200 components for 6d data) it never converges. This results also in all the means and covariances of the components to be nan. Tried different batch sizes and different initializations but nothing worked.

print(gmm.model_.means, gmm.model_.covariances)

-> nan Tensors

[AndreasGerken]

Just found the solution. It was already answered in the issues but it wasn't documented. The issue has to do with a too small covariance_regularization. It can be increased to avoid this issue.

GMM(..., covariance_regularization=1e-5) # Default value is 1e-6

Should to the trick. For some of my data i even had to decrease it to 1e-4.

This behavior should be documented or even returned as a warning. Original issue https://github.com/borchero/pycave/issues/3