Small issue with batchSize>0 and save_latest_freq

[Quasimondo]

I just noticed that if you increase the batchSize to a value > 1 it can happen that depending on the value of save_latest_freq the saving of the lastest_net snapshot is never triggered since the modulo value that checks the counter variable never gets to 0.

[rremani]

Have you tried solving it?

[Quasimondo]

One way to solve it is to use a separate counter and instead of doing a modulo check rather do a secondCounter >= saveFreq check and resetting the counter after saving the snapshot..

[rremani]

Thanks, ill try that ..have you got any improvements after changing the batch size.

[Quasimondo]

Something I am currently doing for testing things quickly is to reduce ndf and ngf from 64 to 32. This allows me to increase the batch size and the visible learning progress is much faster. Of course it comes for the price that the models are limited in their ability to generate finer details since their "pattern library" is limited (at least that's how I explain it to myself).

Another technique I use is to start training my models on a smaller subset of my training data (something like 200 - 300) pairs. The idea here is that this way it learns the basic filters which usually are very similar anyway. Once it starts converging I add more pairs and restart the training using continue_train. Of course it totally overfits on the initial training examples, but from what I can see it seems to work and is much more satisfying to watch :-).

[rremani]

That's quite exciting..m trying to solve the problem of automatic digitisation of scanned images by converting them into cad generated raster outputs. So my image pair consists of a scanned image and the same vector jpg image of the scanned image. But both are high resolution images with min 5000 pixels dimensions, so I am playing around with loadsize, finesize, batch_size and learning rate. If I can overfit the model that would be a good starting point, but as of now it is not converging.

[Quasimondo]

So if I understand it correctly you are trying to sharpen details? In that case I would create lots of small tiles that show just parts of the huge image but at 100% scale. Allow for some padding so you can use jittering. But 256 fineSize shoud be enough, maybe even less.

When you then process the actual data of course you will also have to split it into tiles and later join them back together. In that case you should use overlapping windows since pix2pix tends to produce artifacts at the borders.

[rremani]

I think that can be a good idea. Will work on it and share my results..thanks!