VincentStimper
Optimal hyperparameters and optimal flow design
Hello! Thank you for excellent framework!
For my bachelor thesis I have to adapt SAILOR (Link to project: https://github.com/malicd/sailor). I should implement NF instead of Gaussian Mixture Model. Short about SAILOR: SAILOR is unsupervised algorithm for domain adoption. It uses GMM to compress big dataset and to train the model on target dataset.
How I used NF in SAILOR:
Architecture of transformation/flow:
Number of flows = 4
I am not sure if the number of hidden layers is optimally chosen or not?
number of epochs = 10 000 learning rate = 0.000002
Problem: I have tried to design flows in different ways (flows = 32, 16, 8, 4; changing lr and epochs many times, diff. number of hidden layers, different number of units in hidden layers and so on), but no version gave me good results. During the fitting the model in data I have most of the time case that my loss value is "inf" (or "Nan"). A value for loss starts from 365 and decreasing, and in one moment it starts to print "inf" or "nan". I have also printed loss in diagram (number of epochs = 15 000) and at around 2500 and 3000 epoch the diagram shows two very big spikes, after it continues to decrease. Question: Is there some standard procedure how to choose hyperparameters (lr, number of epochs) and how to design flows? If no, can you give me advice how to create optimal flows with optimal parameters (lr, epochs)? Is 15 000 epochs too big or I can consider it as normal value for epochs?
Thank you in advance!
I have the same issue seeing these spikes in loss and then stabilizing down. Did you get a chance to find the root case and fix it ?
Hi!
If I remember correctly I have experimented with hyperparameters and found some combination that gives Okay results. If your loss function converges to some value and your model gives good results on target dataset (test dataset) you can ignore that spikes.
Kind regards Milan
On Sat, 4 Oct 2025, 20:27 karanchahal-nv, @.***> wrote:
karanchahal-nv left a comment (VincentStimper/normalizing-flows#69) https://github.com/VincentStimper/normalizing-flows/issues/69#issuecomment-3368463084
I have the same issue seeing these spikes in loss and then stabilizing down. Did you get a chance to find the root case and fix it ?
— Reply to this email directly, view it on GitHub https://github.com/VincentStimper/normalizing-flows/issues/69#issuecomment-3368463084, or unsubscribe https://github.com/notifications/unsubscribe-auth/AO6477JGYQSA3NS6F6ZVDNT3WAGQPAVCNFSM6AAAAACIJJ2HNSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTGNRYGQ3DGMBYGQ . You are receiving this because you authored the thread.Message ID: @.***>