Add Elastic Deformation for KPL

[innat]

This type of augmentation is mostly sued in 2D and 3D image augmentation, especially in medical imaging.

- Source.

TensorFlow (DeepMind): How to use the On-the-Fly Elastic Deformation for TensorFlow

[innat]

Some starters in python. https://gist.github.com/fmder/e28813c1e8721830ff9c

import numpy as np
from scipy.ndimage.interpolation import map_coordinates
from scipy.ndimage.filters import gaussian_filter

def elastic_transform(image, alpha, sigma, random_state=None):
    if random_state is None:
        random_state = np.random.RandomState(None)

    shape = image.shape
    dx = gaussian_filter((random_state.rand(*shape) * 2 - 1), sigma, mode="constant", cval=0) * alpha
    dy = gaussian_filter((random_state.rand(*shape) * 2 - 1), sigma, mode="constant", cval=0) * alpha
    dz = np.zeros_like(dx)
    x, y, z = np.meshgrid(np.arange(shape[1]), np.arange(shape[0]), np.arange(shape[2]))
    indices = np.reshape(y+dy, (-1, 1)), np.reshape(x+dx, (-1, 1)), np.reshape(z, (-1, 1))
    distored_image = map_coordinates(image, indices, order=1, mode='reflect')
    return distored_image.reshape(image.shape)

img = cv2.imread('sample.png')[:,:,::-1]
out = elastic_transform(img, alpha=1000, sigma=7)

[bhack]

We could take a look at: https://github.com/tensorflow/addons/blob/v0.15.0/tensorflow_addons/image/dense_image_warp.py https://github.com/tensorflow/addons/blob/v0.15.0/tensorflow_addons/image/sparse_image_warp.py

[Tony363]

@innat is there a style guide or API method guide in which i should implement this feature to? Like an overall API architecture design? I hope to follow up on this and look into what i can do about this after my midterms, after 2-3 days. @bhack mentioned to like at tensorflow_addons implementation. It can't be as simple as reimplementing it to KerasCV API right?

[innat]

is there a style guide or API method guide in which i should implement this feature to? Like an overall API architecture design?

I'm not sure what you're looking for. For style guide or API method guide, is this sufficient? For implementation references, you can follow the following reference implementation.

• ElasticTransform - by albumentation.
• ElasticTransform - by Img_Aug
• ElasticTransform - by Korina
• ElasticTransform - by MONAI
• https://github.com/gvtulder/elasticdeform#tensorflow-wrapper

[Tony363]

I was just thinking if there are any strict coding style I should follow or any KerasCV API design pattern i should keep in mind during implementing the different object classes and methods. I think this is perfect ! I'll be proactive, keep being curious and continue to communicate ~

[Luvideria]

Hello, I recently posted an issue on tensorflow addons: https://github.com/tensorflow/addons/issues/2733

I created a repo containing and explaining a working implementation of the catmull-rom (quite similar to Lanczos-2) interpolation as well as a comparison and a test https://github.com/Luvideria/tensorflow-dense-warp-catmull-rom

It behaves just like dense_image_warp.py but uses a better interpolation scheme. I have no idea if it is up to the standards of these repositories. I use it in my research, and it works well enough.

[innat]

@Luvideria I didn't try dense_image_wrap from tensorflow addons. Do you think that function or your catmull-ran address this feature request (elastic deformation)?

[Luvideria]

@Luvideria I didn't try dense_image_wrap from tensorflow addons. Do you think that function or your catmull-rom address this feature request (elastic deformation)?

I am not 100% sure. My use case is not elastic deformation but motion vector reprojection.

Earlier @bhack mentionned dense_image_warp which may or may not make fully sense for the specific use case of this thread.

I think that the dense_image can help with the last stage of elastic deformation. Dense_image_warp needs a list of sample points, which means the transform has to be computed by other means. Inside dense_image_warp it's like this:

# flatten the input list
       interpolated = interpolate_catmull_rom(image, query_points_flattened)
       interpolated = tf.reshape(interpolated, [batch_size, height, width, channels])

It's just a list of sample points.

In any case, I think that this dense image warp with catmull rom (or higher order, and not necessarily my implementation) deserves to be integrated in one way or another. I can eventually post this somewhere else where you deem it appropriate. I already mentioned it in tensorflow_addons: https://github.com/tensorflow/addons/issues/2733 This seemed appropriate at the time as the original dense_image_warp comes from there.

[bhack]

I think that the dense_image can help with the last stage of elastic deformation. Dense_image_warp needs a list of sample points, which means the transform has to be computed by other means.

If we see tensorlayer elastic_transform and elastic_tranform_multi it makes sense.

https://tensorlayer.readthedocs.io/en/v2.2.4/_modules/tensorlayer/prepro.html#elastic_transform

@Luvideria Probably you could open a new ticket in this repo for porting/refactoring the dense_image_warp from TF Addons eventually with your new interpolation.

Then we could use it for build up the Elastic Deformation (as Tensorlayer done with scipy map_coordinates).

[innat]

@Tony363 @Luvideria Here is another tf implementation reference for elastic transform. Placing it here in case you're interested to contribute.

https://github.com/hirune924/imgaug-tf/blob/main/imgaugtf/augmentations.py#L253-L269

[innat]

@Luvideria Could you please follow @bhack's previous comment? It's needed to port tfa..dense_image_warp layer (with your addition) first as elastic transform might depend on it.

[Luvideria]

@Luvideria Could you please follow @bhack's previous comment? It's needed to port tfa..dense_image_warp layer (with your addition) first as elastic transform might depend on it.

done