Cropping of source image by affine transform

[Nick-Eagles]

Hello,

I created a jupyter notebook demonstrating behavior of STalign that was unexpected to me: after applying an affine transform to the source image in a source-target pair, the pixel locations of the result appear to be using the coordinate system of the target image. In my case, since there is a partial overlap between the Visium sections (and mutually exclusive areas in each), this results in a large piece of the source image being cropped out, making it impractical for use downstream. Another unexpected result is the blurring effect seen at the bottom of the transformed source image.

For my use case, I was hoping to integrate both spot coordinates and images; in particular, with a bit of custom code in addition to STalign, I wanted to create a merged image containing the union of pixels from the source and target. I was imagining that applying an affine transform to the source image would really just rotate the result (similar to how Image.rotate works from PIL), ignoring translations that would otherwise put the area of interest "out of frame" in many cases (like mine).

Perhaps a new parameter to STalign.transform_image_source_with_A() could be added to output an image this way? I understand this is a large request, but would love to hear thoughts about how this issue might be addressed (or other options I could explore).

I'll attached files required to reproduce the notebook's output.

Thanks! STalign is quite a useful tool, so I appreciate your work to develop and maintain it.

Best, -Nick

[Nick-Eagles]

Apologies, I thought GitHub allowed sharing of small files of any type through issues. I can send the image and landmark files upon request (through email?) if that's helpful.

[lcolladotor]

Can you put the files on a gist? https://gist.github.com/ That might be the way to go here. Or share them via a public LIBD OneDrive link.

[Nick-Eagles]

Here are the input files

[JEFworks]

Dear Leo and Nick,

Thanks for the well documented reproduction of the issues.

@mmganant can you please take a look into this as it related to brain?

In general, note that A = STalign.to_A(torch.tensor(L),torch.tensor(T)) should produce a 3x3 transformation matrix. You can simply apply this matrix using matrix multiplication to your pixel coordinates to derive transformed pixels.

Best, Jean

[mmganant]

Hello Leo and Nick, Thanks for using STalign and your detailed description. I believe these issues can be solved by simply padding the images -- I have edited your Jupyter notebook to show you how I would do it: https://gist.github.com/mmganant/123d4ac779c8b02fa816391488b20718

Hope this helps and let me know if you have further questions/clarifications! Best, Manjari Anant

[Nick-Eagles]

Hi Manjari and Jean,

Thanks so much! This does seem like a good workaround. I appreciate the full code example, which makes the solution extremely clear. This answers everything I was wondering about!

Best, -Nick