Set align_flag=True yet correlation templates appear misaligned after register_multisession()

[akatav]

Hi @epnev thanks for your response to my previous issue. i did set the align_flag to True but it does not seem to help, unfortunately. maybe i am missing something.

Just a bit of background: we have 6 sessions images obtained on different days from freely moving mice. there is some shift between these images possibly due to head movement on different days. Individual sessions are motion corrected using caiman moco functions. There are some blood vessels that are imaged as well and using them as landmarks, we can clearly see that the correlation images from the different sessions obtained right after the calls to register_ROIs() (we experimented with/without setting "use_opt_flow") are not aligned well to each other. We also set the shifts_opencv=False as setting it to True gives an empty xy_grid value causing error.

Perhaps i am wrong, but is it possible that the templates updated in register_ROIs() need to be passed down to register_multisession() ? that is, the tempate1 and template2 updated inside register_ROIs() should be saved inside corresponding indices of 'templates' array in register_multisession() ?

Originally posted by @akatav in https://github.com/flatironinstitute/CaImAn/issues/764#issuecomment-641818037

[epnev]

@akatav The aligned templates are not being saved or given as an output from the registration functions. However they're being taken into account when the registration is being performed. The goal is not to align the templates. The goal is to determine which neurons appear in multiple sessions.

[akatav]

@epnev yes, true, i agree. But i think that by setting use_opt_flow=False, template2 passed to register_ROIs is modified (after returning from tile_and_correct()). i plotted these (modified) templates (template2) after the call to register_multisession finishes and noticed that subsequent frames do not align well. Plotting the neurons extracted from every template onto the last template(as background), it seems that only the neurons belonging to the last template contain cell bodies. The other neurons from previous templates seem to point to empty cell bodies only. On the other hand, plotting neurons belonging to every session with the background set to the corresponding template for that session, the above problem does not seem to happen.

[epnev]

I think I understand what you're saying. The code does not currently return the relative shifts between the templates from different days. Essentially you want the variables x_remap, y_remap between all the pairs from the consecutive sessions as calculated here, and you can then take the cumulative sum to obtain e.g., the mapping between the first and the last session and also apply it to the ROIs. I cannot do this at the moment, but it shouldn't be hard if you want to try it yourself. Note, that the result of the registration will not be affected at all, you'll just get access to the details of the process.

Also in general, I've noticed leaving use_opt_flow to its default True value leads to better results.

[akatav]

@epnev thanks.So, each x_remap and y_remap inside register_ROIs() is cumulatively added up and the final cumsum applied to the last template. However, this seems to result in a black background!

The backgrounds of the 2 images are the respective templates of the 2 sessions. Now, the 2 cells inside the circled region (with green pen) are obviously the same cells. My concern is: Will caiman track these cells as one and the same (OR) mark them as different cells owing to the obvious misalignment between the background images ? Because, when plotting on a common background (like, the last session's template), those 2 groups of cells do NOT overlap. Is there a way to label each cell number in the plt.contour() ?

[epnev]

@akatav I had an error in my previous suggestion. What you need to accumulate over sessions are the variables shifts_x, shifts_y as shown here, and not x_remap, y_remap because x_remap, y_remap also contain the grid coordinates and adding those will push you out of the grid which is probably what causes the black background that you observed. Hope that helps.

Is there a way to label each cell number in the plt.contour() ?

You can set display_numbers=True and that will print a number next to each contour.

[akatav]

@epnev thanks, but i think that shifts_x,shifts_y are downsized (not the same resolution as the templates) and hence remapped to the original resolution using x_grid and y_grid. is it possible to post a short code snippet to do this cumsum?

[akatav]

@epnev does display_numbers=True print the same number for the cell as returned by assign_forw matrix that tracks cells across the sessions ?

[EricThomson]

@epnev thanks, but i think that shifts_x,shifts_y are downsized (not the same resolution as the templates) and hence remapped to the original resolution using x_grid and y_grid. is it possible to post a short code snippet to do this cumsum?

@akatav did you ever figure out a simple way to do this? It looks like you have thought about this already, so I'm wondering if you ever found a good solution. I think it would be really useful for visualizing results and triangulating the output of multisession registration.

Barring someone else having already done this ( 🥇 ) I'm thinking not of shifting templates around (that seems hard), but instead to shift contours around and keeping templates fixed. Basically reverse the shift done to the contours (for instance in session 2 undo the shift to match them to template 1, and do the same thing to the transform to session 1 contours). This at least for purposes of visualization will create a useful tool.

I also realize this doesn't quite get at your original issue, but I think that solving that is just hard.