How to use ImageLoss and VolumeLoss?

[lidaiqing]

Hi Jean,

Thank you for building up this awesome library! I would like to do MMD loss for comparing images, do you know how to do it? I saw there are ImageLoss and VolumeLoss in the documentation, do you plan to implement them? If not what is the workaround?

Best, Daiqing

[jeanfeydy]

Hi @lidaiqing ,

You're welcome! The ImageLoss and VolumeLoss layers are still WIP: I have drafts that work fine on my machine, but cleaning and documenting everything takes a lot of time and I'm too busy writing my PhD thesis at the moment. That's unfortunate, but I hope to include them in GeomLoss for October.

As for your second question, fortunately, MMDs are super-easy to implement on images. If A and B encode two densities that you wish to compare, the kernel loss "< A-B, k * (A-B)>" is nothing but the (standard) scalar product between A-B and a "blurry" version convolve(k, A-B), that can be computed using either a PyTorch convolution layer or an explicit Fast Fourier Transform. All in all, you just have to pick a sensible convolution kernel k. In image processing, people tend to favor Gaussian kernels because they can be implemented using separable filters along the 2 or 3 dimensions of images/volumes (see Gaussian blur on Wikipedia or scikit-image). However, from a geometric perspective, it's also one of the worst choices that you can make (as explained in the "gradient flow" tutorials of the GeomLoss gallery). As a robust baseline, if you're only interested in densities that sum up to one, I would strongly recommend to use the "Energy Distance" kernel, which is encoded by a convolution filter that is equal to "minus the distance to the origin". It is much more expensive to compute (with a filter that is at least as large as the input image), but is scale-invariant and generally performs better in measure-fitting tasks. Anyway, in two or three months, all of this (+ all the OT-related stuff) will be easily accessible through the ImageLoss and VolumeLoss layers :-)

I hope that all this answers your question! Best regards,

Jean

[ocourtin]

Hi @jeanfeydy

I hope first your thesis redaction is performing as well as possible...

And as you replied that ImageLoss and VolumeLoss are WIP, could you publish them anyway -as is- ?

It would to give a chance to others folks (e.g me or @lidaiqing, or others) to help you to finish to clean it...

Or at the very least to beta test...

The works you already done is really impressive, and i'm indeed really curious to compare what we can extract with OT losses, rather than classical ones (as Lovasz) from Imagery...

Best,

[jeanfeydy]

Hi @ocourtin ,

Thanks for your feedback!

Unfortunately, I am extremely late on my thesis now and cannot spare the ~5 days that would be needed to clean up my drafts and put them on the repo, even for a beta release. The whole thing can get very technical (see e.g. Bernhard Schmitzer's work ), and putting online a half-baked version would be a waste of time for everyone.

Note, however, that if you want to do some fast test and prototyping, you can simply represent your density maps (image or volumetric) as weighted point clouds, and run SamplesLoss on the encodings. This is most efficient for small masks (e.g. with <100k non-empty voxels), and should be good enough in many cases: see for instance the 2D tutorial.

I hope this will be good enough for you: at the very latest, everything should be up by January 2020 :-)

Best regards,

Jean

[JunMa11]

Dear @jeanfeydy,

Thanks for sharing the great work. I hope everything goes well with your PhD thesis.

Anyway, in two or three months, all of this (+ all the OT-related stuff) will be easily accessible through the ImageLoss and VolumeLoss layers :-)

Are there any updates on ImageLoss and VolumeLoss layers? I do not find this in this repo.

Looking forward to your reply.

Best regards, Jun