aicsimageio
aicsimageio copied to clipboard
AllenCellModeling
Mosaic merge of LIF
System and Software
- aicsimageio Version: 4.0.3
- Python Version: 3.9.6
- Operating System: MacOS Big Sur 11.4 (but everything on Apple Silicon M1 arm64 native)
Description
Reading a mosaic merge from LIF generates a mismatched image. It appears that the image is, for example, instead of x=4 y=3 merged as y=4 x=3 Example LIF scene "R1" (TileScan 2/B/2/R1) https://www.dropbox.com/s/jk9se8i1kpzqvsn/20210428_24w_L929_Ho_B2C3.lif?dl=0 For comparison, scene "R1_Merged" (TileScan 2/B/2/R1_Merged) is the mosaic merge by LAS X Note: I'm not sure the overlap is taken into account?
Expected Behavior
Generated a proper merge automatically or provide an option to provide the merging pattern and overlap
Reproduction
Using: https://www.dropbox.com/s/jk9se8i1kpzqvsn/20210428_24w_L929_Ho_B2C3.lif?dl=0
from aicsimageio import AICSImage
img = AICSImage("20210428/20210428_24w_L929_Ho_B2C3.lif")
img.set_scene("R1")
img.dims
img.set_scene("R1_Merged")
img.dims
Environment
As mentioned above, I have native Apple Silicon M1 arm64 python and aicsimageio
I've figured out the problem—I think. I'm going to outline my observations and thought process for future reference for anyone interested—I'm not yet sure how to really do this in python and integrate a fix into AICSImageIO.
Anyhow, as far as I can tell the current approach in AICSImageIO doesn't really use the information from readlif and makes some assumptions which aren't always met. Here's how I understand it:
readlif returns mosaic_positions which is a list of tuples: (FieldX, FieldY, PosX, PosY)
- FieldX, FieldY are tile coordinates as column, row indexes
- PosX, PosY are the tile positions in physical coordinates (in meters, not pixels)
Importantly the list is in the correct tile order—more on this later. So for example, just giving two rows worth of FieldX, FieldY of my LIFs:
(0,0), (1, 0), (2, 0), (3, 0), (3,1), (2,1), (1,1), (0,1) ...
We can see this is a zig-zag acquisition, the default in our LAS X.
What AICSImageIO does is figure out the number of columns and rows to get the basic mosaic shape (ny, nx): https://github.com/AllenCellModeling/aicsimageio/blob/57e83755d8beb1dbbb880a4d3b4657770812d36d/aicsimageio/readers/lif_reader.py#L581-L586
where:
last_tile_position = selected_scene.info["mosaic_position"][-1]
BUT: This works only if the last position of mosaic_position, the last tile imaged, is really at the ny, nx position. For a zig-zag tiling (the default in our LAS X) this may not be the case for an even number of rows. See this sequence:
(0,0), (1, 0), (2, 0), (3, 0), (3,1), (2,1), (1,1), (0,1)
which would set last_tile_position to the tile (0,1) and return wrong ny, nx values.
Let's add another row to make it an odd number: (0, 2), (1, 2), (2, 2), (3,2)
Now, last_tile_position is the correct tile and we get (ny, nx) as (4, 3). We have 4 cols, 3 rows, so that matches.
BUT: When stitching, the dimensions of the mosaic end up wrong, because I think row/col are mixed up here: https://github.com/AllenCellModeling/aicsimageio/blob/57e83755d8beb1dbbb880a4d3b4657770812d36d/aicsimageio/readers/lif_reader.py#L537-L541
the range of ny is 4, the number of columns (obtained from FieldX in mosaic_position), while nx is the number of rows (obtained from FieldY). So for the example above, the mosaic prepared by AICSImageIO ends up as 3 col by 4 row.
For my (x:1920, y:1440) images and a 4 col, 3 row mosaic, I get square mosaics from AICSImageIO stitching: (YX):(5757, 5758) which corresponds to: (y:1440x4=5760, x:1920x3=5760). LAS X stitching of the same tiles:(Y,X)(3899, 6811) vs. the expected: (y:1440x3 = 4320, x:1920x4=7680). This reflects the overlap in tiles, more on this later.
Further, when placing each tile in the mosaic, in other words selecting the image from the list indexed by M and putting it in a given (col, row), M is computed as:
(row_i * nx) + col_i
Again this will only work for L-R, row-wise and not zig-zag. For the example above this yields:
M: [0, 1, 2, 3, (3+3=6), (3+2=5),(3+1=4), (3+0=3)]
So to solve this, I think it's probably best to get the mosaic_position list from readlif and iterate over that, because the index of the list is the proper M index for that X, Y tile position.
Also, one should be able to use the physical coordinates PosX and PosY to account for tile overlap.
First, use the values of the PosX, PosY of the (0,0) tile to get the origin and make the other tile Pos be relative values to that origin. One can rescale back to pixels to make life easier. Then the position delta between tiles relative to the tile size will reflect the overlap. First pass, one could just replace the pixels in the overlap region with those from the next tile, but some blending could potentially also be implemented in the future to mimic what LAS X does (see: https://imagej.net/plugins/image-stitching).
Yea, we need to do a pass on all of our mosaic stitching for CZI and LIF again.
Fully agree that we should use the bounding box info and not just the indices.
We have added a new maintenance feature to clean up stale issues and PRs. Adding this comment to set a baseline for ‘Stale’
Hi @psobolewskiPhD, I opened #480 to provide a potential avenue to solve this issue. I agree with you that this package is swapping the X and Y coordinates by default and that seemed to have messed with your image. However, even after turning that off I seemed to be unable to match up with the expected image, so I enabled flipping the X and Y coordinates (which results in a 180 degree rotation) too which seemed to make it line up. If you get the opportunity I'd appreciate you taking a look at my pull request to see if that aligns with your thinking and may solve this issue.
I did not address improving the stitching of overlapping tiles using the physical coordinates. Is that something that would need to be fixed to consider this ticket done?
Sorry, been busy in RL.
To be clear, as far as I was able to tell (https://github.com/AllenCellModeling/aicsimageio/issues/278#issuecomment-922492259) it's not just that X and Y were being swapped but that tile positions were based on a faulty assumption of row-by-row scanning.
Anyhow, I don't think that a solution to his issue needs to deal with overlap. Just returning rough mosaics that have the tiles place properly would be a good fix—sometimes overlap may be really low or even 0.
I think the key is handling the actual tile locations in a way that accounts for zig-zag acquisition which is the default on our Leica 'scope.
I think that cropping tiles to deal with overlap certainly could/would be a potential followup?
PS. I will try and take a closer look at the code/PR ASAP, but don't let me block you!
@psobolewskiPhD Okay sounds great! This was closed as part of my PR being merged, but feel free to re-open or create another ticket if you find the fix unsatisfactory for your use case!