Sample explorer with z stack + multiple points

[CedricEsp]

Hello (again),

I wonder if there is an elegant way to couple the multi-D Acquisition with the sample explorer? For example, if one wants to acquire 4x4 tiles with z stack at multiple positions.

Thanks!

[tlambert03]

Hey @CedricEsp! I don't think this is immediately possible, though @fdrgsp tells me he has a branch to this effect. We'll try to make this possible

[CedricEsp]

Thanks! In the meantime I tried different option without a lot of success: My first though was to loop through the different position and start a mda but I ended up with the following error which was pointed out somewhere else:

ValueError: Cannot start an MDA while the previous MDA is still running.

code is like that:

for i, pos in enumerate(self.xyz[:,0:2]):
      sequence = MDASequence(
      channels=seq_ch,
      z_plan={"range": z_range, "step": z_step},
      axis_order="tpcz",
      stage_positions=[(pos[0], pos[1], z_AF)]
      )

     self.main_window.mda.set_state(sequence)
     self.main_window.mda._on_run_clicked()

I tried to add a self.core.mda.cancel() at the end, it does go through the all loop but nothing appears in napari.

My second though was to build the MDA "myself" (more like a bad copy of what is done): But, probably because it's a bad copy and I am not exactly sure of what I am doing it doesn't work as expected (mainly the data are not "streaming" into napari but more opening when the acquisition is done:

@ensure_main_thread
def temp_file(core, xyz, w3, z_stack_pos):
    tmp = tempfile.TemporaryDirectory()
    data = np.empty([len(xyz),len(w3.value), len(z_stack_pos), 512, 512])
    dtype = f"uint{core.getImageBitDepth()}"
    shape = data.shape
    tp_viewer = zarr.open(str(tmp.name), shape=shape, dtype=dtype)
    layer = v.add_image(tp_viewer, blending="additive")

@ensure_main_thread
def nd_ac(core, xyz, w3, z_stack_pos):
    for l, xy_pos in enumerate(xyz[:,0:2]):
        core.setXYPosition(xy_pos[0], xy_pos[1])
        for m, ch in enumerate(w3.value):
            core.setConfig("Channel", ch)
            core.setExposure(30)
            for n, z_pos in enumerate(z_stack_pos):
                core.setZPosition(z_pos)
                core.snapImage()
                tp_viewer[l,m,n, :, :] = core.getImage()
                time.sleep(10)
                v.dims.set_point(0,l)
                v.dims.set_point(1,m)
                v.dims.set_point(2,n)

Any advice would be more than welcome :)

[ianhi]

You're close! But rather than rebuilding the MDA engine code you should go with the first approach where you construct a useq.MDASequence. The key insight is that rather than starting multiple MDAs you should construct one big MDA with multiple positions. Multiple MDAs aren't allowed because on real hardware this could cause serious issues when you have conflicting commands.

So in your case (If I assume that self.xyz is an Nx3 array and z_AF is a float) I would create an run an MDA like this:

pos = np.asarray(self.xyz)
pos[:, -1] = z_AF

from useq import MDASequence
sequence = MDASequence(
    channels=seq_ch,
    z_plan={"range": z_range, "step": z_step},
    axis_order="tpcz",
    stage_positions=pos
)

also, rather than interacting with the GUI via code I'd suggest directly interacting with the underlying object controlling the hardware. That lives in pymmcore-plus and we have an example of how to use it with this project here: https://pymmcore-plus.readthedocs.io/en/latest/examples/napari-micromanager.html

so in your case instead of programmatically clicking I would do the following:

from pymmcore_plus import CMMCorePlus

core = CMMCorePlus.instance() # get the global singleton. i.e. the same one used by napari-micromangaer

core.run_mda(sequence)

[CedricEsp]

@ianhi thank you so much for your answer! I realized that I forgot a (very) important part. Every nth position I run an autofocus (image base AF). So z_AF is actually a new focus position: z_AF = self.autofocus(ch_AF = ch_AF, z_range_AF=z_range_AF, z_step_AF=z_step_AF)

Thank you for your advice on not programmatically clicking.

[ianhi]

@ianhi thank you so much for your answer! I realized that I forgot a (very) important part. Every nth position I run an autofocus (image base AF).

Ahhh I see. I think long term we want to incorporate a way to define feedback loops like this in useq-schema https://github.com/tlambert03/useq-schema/discussions/28 however there's not currently a built in way to do this.

I think the easiest strategy for you is actually to extend the default MDA engine to add in this autofocus step. I'll give some links to examples of this below but I think first a a quick explanation of how the various layers interact is in order:

The overall diagram of interactions looks like this:

You should primarily (perhaps entirely) interact with the GUI via the mouse and keyboard, but spawn it from a python script. Any actions you take via core.___ methods will be relayed to napari-micromanager via the signalling systems that pymmcore-plus adds on top of pymmcore.

For working with MDAs you should interact with the signals from the "MDA Engine" accessible as core.mda. The engine is the code that actually runs the acquisition loop (move to position, snap an image etc). See more about it here: https://pymmcore-plus.readthedocs.io/en/latest/examples/mda.html#multidimensional-acquisition-mda

An important design choice is that the built in engine isn't set in stone, there are only two hard and fast rules:

1. Only one engine can be registered with the core at any time
2. all engines must conform to the PMDAEngine protocol

So in your case I would suggest that you make your own engine that subclasses the default engine and adds your autofocus behavior. Unfortunately we don't really have any documentation on how to do this but I can point you to some examples that i've made, both test cases and a real engine I use to control a laser on a real scope.

The basic pattern is:

from pymmcore_plus.mda import MDAEngine
class myEngine(MDAEngine):
    # modify the methods most relevant to you (e..g prep_hardware)
    ....

engine = myEngine()
core.register_mda_engine(engine)
core.run_mda(sequence)

So I suggest to subclass the default engine because then you can make use of the helpful methods like _wait_for_event so you only have to re-write the run method. You can see the default engines methods here: https://github.com/tlambert03/pymmcore-plus/blob/112519604603ae369adfb80ba2fe789786266249/pymmcore_plus/mda/_engine.py#L42

Other examples:

See this package for nice development engines that will make use of my mda-simulator package: https://github.com/ianhi/pymmcore-MDA-engines

in particular the drift correction engine may be of interest: https://github.com/ianhi/pymmcore-MDA-engines/blob/4d850ffe884f3323a319c49f8ee24c0edf3427b0/pymmcore_mda_engines/_engines.py#L103 this corrects for XY drift, but I would love to have an example of image based Z drift correction.

I'd also look at the run method of my raman_mda_engine which I use to collect raman spectra of cells only for certain images (middle of the BF Z stack): https://github.com/ianhi/raman-mda-engine/blob/3514b261dd4739a6b71caeb6c9f447e43b9fd8a5/raman_mda_engine/_engine.py#L135

[ianhi]

You may also find parts of this repo's launch-dev script: https://github.com/tlambert03/napari-micromanager/blob/main/launch-dev.py helpful. It automatically installs the mda-simulator demo camera and such. I tend to copy part of that into a jupyter notebook when im developing these things

[CedricEsp]

Thank you @ianhi for taking the time to give me such a detailed answer. If I understand well..🤯..I think it's exactly what I need.

[ianhi]

If I understand well..exploding_head.

It's definitely a lot to take in - as it stands you may be the first person not involved in writing the libraries to try to extend the MDA engine. So feel free to ask quesitons and try to keep track of what is confusing for you so we can use this experience to craft good documentation for the next person who has a go at it!

I think it's exactly what I need.

This is the option that is most difficult to do but also provides the most power. It should be possible to do arbitrarily complex things using this so definitely encompasses your needs. The tricky library design part is figuring a way to provide an easier solution.

Feel free to ask questions as they come up and good luck!

[tlambert03]

thank you @ianhi!

[CedricEsp]

@ianhi I tried a very simple version of the 2 examples you gave me, it seems that there is no more communication with Napari-micromanager (if I manually run the sequence in the napari GUI it works fine) but if if I run after registering the engine image don't update anymore in napari.

class AutoFocus(MDAEngine):
    def __init__(
        self,
        mmc: CMMCorePlus = None,
    ):

        super().__init__(mmc)


    def run(self, sequence: MDASequence) -> None:
        """
        Run the multi-dimensional acquistion defined by `sequence`.
        Most users should not use this directly as it will block further
        execution. Instead use ``run_mda`` on CMMCorePlus which will run on
        a thread.
        Parameters
        ----------
        sequence : MDASequence
            The sequence of events to run.
        """
        self._prepare_to_run(sequence)
        for event in sequence:
            if event.index.get('p') in [0,4,10]:
                event.z_pos += 100
                self._prep_hardware(event)
        self._finish_run(sequence)

With using this for simulator:

import napari
import numpy as np
from useq import MDASequence
from pymmcore_plus import CMMCorePlus
from pymmcore_plus.mda import MDAEngine

try:
    from mda_simulator.mmcore import FakeDemoCamera
except ModuleNotFoundError:
    FakeDemoCamera = None

v = napari.Viewer()
dw, main_window = v.window.add_plugin_dock_widget("napari-micromanager")

core = CMMCorePlus.instance()
core.loadSystemConfiguration("../test_config.cfg")

Here is the sequence:

sequence = MDASequence(
    channels=['FITC'],
    z_plan={"range": 60, "step": 5},
    axis_order="tpcz",
    stage_positions=list(map(tuple, xyz))
)

main_window.mda.set_state(sequence)

And here is how I run it:

engine = AutoFocus()
core.register_mda_engine(engine)
core.run_mda(sequence)

[ianhi]

but if if I run after registering the engine image don't update anymore in napari.

Are you running the code before or after napari has launched? If you run it before a napari.run() then the right hooks won't be set up yet.

[ianhi]

oops nevermind it's much simpler than that. In your run function you don't ever snap an image! Also you should make sure prep_hardware runs for every event. Overall it should look like this:

        self._prepare_to_run(sequence)
        for event in sequence:
            if event.index.get("p") in [0, 4, 10]:
                event.z_pos += 100
            # CHANGE: I also de-indented this line so it runs for every event
            self._prep_hardware(event)

            # ADDED LINES BELOW
            self._mmc.snapImage()
            img = self._mmc.getImage()
            self._events.frameReady.emit(img, event)
           # END OF ADDED LINES
        self._finish_run(sequence)

one final point. After doing pip install mda-simulator you should add this code block:

if FakeDemoCamera is not None:
    # override snap to look at more realistic images from a microscoppe
    # with underlying random walk simulation of spheres
    # These act as though "Cy5" is BF and other channels are fluorescent
    fake_cam = FakeDemoCamera(timing=2)
    # make sure we start in a valid channel group
    core.setConfig("Channel", "Cy5")

[CedricEsp]

Easy enough, I guess I misunderstood how it works, you actually "recreate" the all engine, it's pretty awesome you added that, it makes (relatively) easy to create a complex acquisition workflow (although I am pretty sure I will come back with more questions though...), thanks!

[CedricEsp]

Hello and thanks again for all your help last week.

So some follow up questions:

I need to create overview images at different position - my plan was/is to move the stage to position XY[0], acquire large image, then move to position XY[1] acquire large image etc. Here is the backbone of the large image acquisition engine following your help from last week:

def run(self, sequence: MDASequence) -> None:
        """
        Run the multi-dimensional acquistion defined by `sequence`.
        Most users should not use this directly as it will block further
        execution. Instead use ``run_mda`` on CMMCorePlus which will run on
        a thread.
        Parameters
        ----------
        sequence : MDASequence
            The sequence of events to run.
        """
        self._prepare_to_run(sequence)
        nbr_slide = 3
        
        #Should add in sequence center position of different slides to loop through
        arr = []
        for event in sequence:

            self._mmc.snapImage()
            img = self._mmc.getImage()
            arr.append(img)

        arr_large = self.construct_large(arr)
        
        self._events.frameReady.emit(arr_large, event)
        self._finish_run(sequence)

So the first time I run it, it works great but then if I run it again, it override the older capture and replace it with a multiple position image : Is there a way to change that behavior?

It might be related but how do you actually adjust the name of the layer in napari? At the moment it's automatically set to Exp_....?

Thanks!

[fdrgsp]

Hi @CedricEsp, if you want you can check this PR #180 where I'm adding mda capabilities to the sample explorer.

At the moment, the only change that has to be made in order to make it work is in the useq-schema: https://github.com/tlambert03/useq-schema/pull/44.

It is not perfect yet, I still have few issues solve (e.g. reset_view and viewer.camera.zoom) but the acquisition should work fine.

Let me know if this help!

[CedricEsp]

Thanks @fdrgsp ! I guess it could definitely help, so you are adding one more argument to the MDAsequence? I am still curious, why my option doesn't work though?

[ianhi]

It might be related but how do you actually adjust the name of the layer in napari? At the moment it's automatically set to Exp_....?

Are you using the exact same MDASequence object when you run the mda? If so the names are the same because the names come from the sequence.uid. A fix could be to just regenerate the sequence before running thus giving it a new uid https://github.com/tlambert03/napari-micromanager/blob/de45a8c8bb29ddda65dade08e52e822642c1f484/micromanager_gui/main_window.py#L297 https://github.com/tlambert03/napari-micromanager/blob/de45a8c8bb29ddda65dade08e52e822642c1f484/micromanager_gui/main_window.py#L309

[CedricEsp]

Hi @fdrgsp, sorry for the basic/naive question, I wanted to test your PR but I am not sure what is the best way to install napari-micromanager from source?

[ianhi]

@CedricEsp you can directly install federico's branch with this command: pip install git+https://github.com/fdrgsp/napari-micromanager@multiD_explorer (https://stackoverflow.com/questions/20101834/pip-install-from-git-repo-branch)

but a more general strategy would be to be make your own fork of this repo, do pip install -e ., then set talley's repo as a remote, then you can check out the PR branch. At that point whatever branch you have checked out in your local copy will be the version your code uses.

[fdrgsp]

@CedricEsp also remember to install in your environment this branch from the useq-schema or the multiD_explorer won't work: pip install git+https://github.com/fdrgsp/useq-schema@expose_name_in_MDAEvent.

[CedricEsp]

Thank you for your help, I have it working nicely locally. Here is my question now, I created an engine to be able to run autofocus every nth tile in a workflow (as per @ianhi advice). Is it possible to add an extra key to the MDASequence dictionary, with like a "grid starting position"?

Now I have something like that:

engine = AutoFocus(viewer=viewer, param=param)
core.register_mda_engine(engine)
core.run_mda(sequence)

[CedricEsp]

To add some context to what I am doing (which works fine):

      with open(path+"param.json", "r") as file:
            param = json.load(file)
    # ROIs where created manually by using the points layer in napari   
        coord = np.load(roi)
        for reg in coord:
            XY = list(reg)
            move_load_pos(core=core, XY=XY) 
            sequence = create_sequence(param = param, XY = XY, core=core)
            engine = AutoFocus(viewer=viewer, param=param)
            core.register_mda_engine(engine)
            core.run_mda(sequence)
            # I need to add that otherwise not all stack are loaded to the viewer
            while core.mda.is_running():
                time.sleep(0.1)

I wonder instead of looping through the coordinate I could use the Grid you created @fdrgsp ?

[fdrgsp]

Hey @CedricEsp, how does your AutoFocus engine look like?

[CedricEsp]

    def run(self, sequence: MDASequence) -> None:
        """
        Run the multi-dimensional acquistion defined by `sequence`.
        Most users should not use this directly as it will block further
        execution. Instead use ``run_mda`` on CMMCorePlus which will run on
        a thread.
        Parameters
        ----------
        sequence : MDASequence
            The sequence of events to run.
        """
        self._prepare_to_run(sequence)
        
        for event in sequence:
            #AF every few tiles
            if event.index.get("p") in [0, 4, 10]:
                #run a simple AF and get the new z position before doing a z stack
                pos = self.autofocus()
                event.z_pos =pos
            
            self._prep_hardware(event)
   
            self._mmc.snapImage()
            img = self._mmc.getImage()
            
            #Some saving happening here

            
            self._events.frameReady.emit(img, event)
         
        self._finish_run(sequence)

[fdrgsp]

@CedricEsp sorry, I"m trying to understand...where do you use this?

with open(path+"param.json", "r") as file:
           param = json.load(file)
   # ROIs where created manually by using the points layer in napari   
       coord = np.load(roi)
       for reg in coord:
           XY = list(reg)
           move_load_pos(core=core, XY=XY) 
           sequence = create_sequence(param = param, XY = XY, core=core)
           engine = AutoFocus(viewer=viewer, param=param)
           core.register_mda_engine(engine)
           core.run_mda(sequence)
           # I need to add that otherwise not all stack are loaded to the viewer
           while core.mda.is_running():
               time.sleep(0.1)

Is it to create the list of starting position for the grids (same as adding positions to the Grid Starting Position table in napari-micromanager)?

[CedricEsp]

@fdrgsp I acquire an overview image at low res, then add point on ROI and save these points. (The param.json file is just a dictionary created by the user with the number of tiles, channels, z stack range etc that is use to create the sequence). I then go through the different points/ROIs (you could think about them as the center of the grid) and acquire grid images with z stack and multi channels at higher resolution.

[CedricEsp]

But at the moment I am not using your PR

[ianhi]

@CedricEsp some small suggestions:

1. Only make and register one engine - then pass param to it either a property that can be set: engine.param = new_value or have that be some metadata in the sequence
2. Instead doing a time.sleept loop just do core.run_mda(sequence).join() this will block until it finishes

Overall it could look something like this:

# at start of script
engine = AutoFocus(viewer=viewer)
core.register_mda_engine(engine)


# do some stuff - then load this file
with open(path+"param.json", "r") as file:
           param = json.load(file)

# set the engine param to something other than the default
engine.param = param

coord = np.load(roi)
for reg in coord:
       XY = list(reg)
       move_load_pos(core=core, XY=XY) 
       sequence = create_sequence(param = param, XY = XY, core=core)
       core.run_mda(sequence).join()

[CedricEsp]

Thanks @ianhi, I made the modifications. The stack are only visible in napari when the acquisition is done (every sequence is 16 np, 4 nc and 20nz), I wonder if it's expected? If I don't loop through the coordinate the viewer stream with the acquisition.

[ianhi]

The stack are only visible in napari when the acquisition is done

I suspect this is because you are never giving napari a chance to display the new images because the main thread is always busy - either from the time.sleep or the .join(). Can you combine each reg into one mega sequence? Then you could run without the .join().

Alternatively I think you can start your own thread that runs MDAs in a loop. Essentially borrowing from the run_mda function.

Not tested but I think something like this should work:

from __future__ import annotations
from threading import Thread
import useq


def run_many_mda(mdas: list[useq.MDASequence], core: CMMCorePlus = None) -> Thread:
    """
    run multiple separate MDAs in a loop without blocking the main thread
    """
    core = core or CMMCorePlus.instance()
    if core.mda.is_running():
        raise ValueError("Cannot start an MDA while the previous MDA is still running.")

    def f(mdas):
        for seq in mdas:
            core.mda.run(seq) # this is blocking so don't need to .join()

    th = Thread(target=f, args=(mdas,))
    th.start()
    return th