viz.plot_sensors('3d') doesn't handle mouse clicks to reveal sensor names correctly

[hoechenberger]

MWE:

from pathlib import Path
import matplotlib
import mne

matplotlib.use('Qt5Agg')

sample_dir = Path(mne.datasets.sample.data_path())
sample_fname = sample_dir / 'MEG' / 'sample' / 'sample_audvis_raw.fif'

raw = mne.io.read_raw_fif(sample_fname, preload=False)
raw.plot_sensors('3d')

Clicking on the frontal sensors reveals names of channels someplace else.

https://user-images.githubusercontent.com/2046265/115535014-a6acc000-a298-11eb-881d-bacf661ee57d.mov

When show_names=True is passed, the channels are labeled correctly, though, leading me to believe that the odd behavior described above is related to click -> channel mapping

For completeness, here's a 2D sensor plot:

macOS with Qt5 backend.

[mmagnuski]

Oh, yes, I experienced it too. I think this worked correctly some time ago but I'm not sure.

[hoechenberger]

I remember it being like this for a while now, always forgot to file an issue report…

[drammock]

I played around a bit and have concluded that the event.ind returned by the matplotlib pick event is not deterministic and not stable before/after a click-drag rotation of the axes. Try it out: click on a point that you like, keep your eye on it while you rotate the axes, then click on that point again.

import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
plt.ion()


def onpick(event):
    ind = event.ind
    print(f'index: {ind}')


x = np.tile(np.linspace(0, 1, 5), 3*7)
y = np.repeat(np.linspace(0, 1, 7), 3*5)
z = np.tile(np.linspace(0, 1, 3), 5*7)
c = np.linspace(0, 1, 3*5*7)
s = np.full(3*5*7, 100)

fig, ax = plt.subplots(subplot_kw=dict(projection='3d'))
ax.scatter(x, y, z, s=s, c=c, picker=True)
fig.canvas.mpl_connect('pick_event', onpick)

[hoechenberger]

@drammock Wow, this seems like an actual bug in Matplotlib then… 💩

[drammock]

I'm not sure if they would consider it a bug. efficiently plotting 2D shapes in 3D space is not easy; I'm not surprised there's no guarantee that the points are drawn in the same order every time. I guess maybe you could say it's a bug that they're allowed to be pickable?

[hoechenberger]

I guess maybe you could say it's a bug that they're allowed to be pickable?

Yes, if their index is conditional on axis orientation, even though one is clearly referring to the same entity, then I would say it's a bug. If the index is mutable, it probably shouldn't be exposed in the first place.

Question is, how are we going to deal with this? The current behavior is not great. I suppose we could manually deduce from the current viewport orientation and channel montage which channel the user most likely intended to click on. But it sounds like a lot of work to get right :(

[drammock]

Question is, how are we going to deal with this?

Step 1 is checking if they're already aware of this, and if not, opening an upstream issue.

suppose we could manually deduce from the current viewport orientation and channel montage which channel the user most likely intended to click on.

Something based on solution 5 on this page might work. Want to give it a try?

[larsoner]

I'd open a matplotlob issue first. My guess it that they have (or should have) some form of this logic in place already. In particular they might do the 3d to 2d projection internally, in which case there's potentially a problem with 2d marker or path collection picking. We can do the viewport gymnastics ourselves if we need to, but it would be good to hear them tell us that we should need to first.

[larsoner]

I'm going to push to 0.24 since it seems like it might take a while to figure out the right solution here