comparing two optimizer, adam, lbfgs

[Henry-Yan-HS]

Any insights on the performance/ pros, cons of two optimizers:

Can someone give a brief explanation of how both works, what are their respective advantages and disadvantages, and how they performed?

Thanks!

[darkstorm2150]

a short example is that adam could do larger images at the cost of quality, lbfgs is considered better at generating art, but is limited is resolution because of the high GPU memory footprint.

[cablecutter]

Adam can be useful when using multires or similar scripts that slowly step up the output image size. When Adam has an example init-image from lbfgs the aesthetic is more or less maintained. So, you would step up to your max allowable resolution under lbfgs and then continue with Adam beyond that.

[linrio]

@vic8760 I used the adam to calculate the iamge, actually the quality is lower than that with L-BFGS. The quality of transfered image is coarser.

[linrio]

@cablecutter What's the mean of

... When Adam has an example init-image from lbfgs the aesthetic is more or less maintained ... ?

[ProGamerGov]

The Adam optimizer can be greatly improved in Neural-Style by simply changing it's default parameters to more "stable" values according to my research here: https://github.com/jcjohnson/neural-style/wiki/Fine-Tuning-The-Adam-Optimizer

With the improved parameter values, Adam become a lot closer to L-BFGS in terms of quality.

Another trick you can do to improve quality when using multiscale scale resolution, is to use L-BFGS for the lower image size steps (where most of the major changes occur), and then use Adam for the subsequent larger image size steps.

[nile649]

Hi, I was wondering if anyone could help me with code for replacing lbfgs with Adam. I would be happy because I want to study effects of different losses.