Visualization of single channel annotations

[marsipu]

Is your feature request related to a problem? Please describe. Support for visualization of channel-specific annotations as described in mne-tools/mne-python#8946 and mne-tools/mne-python#8947 and #71 .

Describe the solution you'd like Currently annotations are displayed by use of pyqtgraph's LinearRegionItem. By setting span it should be possible to draw annotations only for single channels. If we want subsegmented Regions, it might be better for performance if we modified the paint-method of AnnotationRegion(which is the currently used subclass of LinearRegionItem) to draw rectangles only where channels are active. So there would be still just on AnnotationRegion per marked timespan.

Also the following could be added:

• [ ] Shift+Clicking on channels inside annotations add/remove them from the annotation
• [ ] A dialog as suggested in #71 to select channels

Describe alternatives you've considered Pyqtgraph's ROI could also be an alternative, but I think it has much more functionality we don't need.

[prlabu]

Anything new on this? I am looking to visualize annotations for subsets of channels, exactly as you describe!

[larsoner]

No I don't think anyone has worked on it yet

[nmarkowitz]

Hi, I'm going to give this a try. Is there someone in the MNE dev team that could potentially give me pointers on how to best do this? I have thoughts but I think I would need to make new properties to store channel information for annotations.

[larsoner]

@nmarkowitz Annotations already has a ch_names attribute, see https://mne.tools/dev/generated/mne.Annotations.html . So really at this point it's mostly an issue of how to actually display and interact with the channel-specific annotations rather than how to store the annotations themselves in an object.

[nmarkowitz]

@larsoner I saw that but I also see that there are many other properties in the BrowserParams object such as new_annotation_labels that I'm unsure if I need to worry about or need to maker corresponding new properties such as new_annotations_channels. My thoughts for how to implement this are (and let me know if you disagree with these)

• While in annotation mode, shift+left-click or shift+right-click on a channel to mark that channel as being associated with that annotation
• Draw a rectangular region around channel name, similar to AnnotationRegion object, to show that channel is associated with that annotation

Any other thoughts or things that should be changed with this?

[larsoner]

there are many other properties in the BrowserParams object such as new_annotation_labels

Not sure why those would need to be there. Naively I'd expect the raw.annotations "just" to be modified on the fly based on interactions. So I'd see how far you can get with that approach (modifying raw.annotations inplace) here first, and only add new stuff when needed (and justify with code comments ideally!).

Draw a rectangular region around channel name

Maybe but to me a simpler approach would be clicking the channel name (probably while holding shift) to add or remove it from the annotation's set of channel names. But really I would start with the simplest minimal implementation that gets the ball rolling. Then people can try and comment before you bother with other implementations. The minimum I'd expect would be something like:

1. Some way to select one annotation to say "this is the one I want to make channel specific" (or select or whatever)
2. Some way to add/remove channels to/from it (easiest is probably clicking the channel name while in channel-editing mode)
3. Some way to exit the channel-setting / selection mode

But I have never dealt with channel-specific annotations so I could be wrong. @nmarkowitz do you? What's the most natural interface for you, building on the current annotation interface?

If we want subsegmented Regions, it might be better for performance...

Just want to note that this is in the top comment, but I wouldn't worry about drawing performance to start. I think we should design a nice usable interface first and get it working the way people want for their workflows. Then we can consider/improve performance in follow-up PRs. To that end I think if we "just" make AnnotRegion smart enough to either draw one monolithic/all-channel LinearRegionItem or multiple LinearRegionItems with spans (with one for each selected channel) it's already a good start.

[drammock]

My thoughts: split this into 2 PRs:

1. ONLY adds visualization of existing per-channel annotations. Creating new per-channel annotations must still be done through code.
2. adds the ability to create per-channel annotations through the GUI

for the second one, I think the interaction design needs some brainstorming. Possible ideas:

• maybe we need a toggle to switch from "all-channels" mode to "specific-channels" mode
• maybe adding specific-channel annotations is always a 3-step process: create an all-channels annotation, convert it to a specific-channels annotation, then specify which channels it applies to (somehow)

[nmarkowitz]

I'll start with some basic. While in annotation mode, shift+click a channel on the channel name or the channel trace will make that channel part of the annotation. I'll attend office hours tomorrow as well if we want to further discuss

[nmarkowitz]

Here's my first mock up of how we could visualize channel-based annotations. The annotation region would still be shown while adding an additional highlight over the specific channels. I also added an additional button to toggle showing channel annotations in the annotation editor. Let me know your thoughts

[larsoner]

I expected the channel-specific annotation to only cover the annotated channel(s) rather than spanning all channels in one color and the annotated channel in a different (darker or lighter) version of that color. To me it makes more sense. And you could actually have two annotations -- one channel-specific, and one non-channel-specific -- covering the same (or very similar) spans of time and this will look the same / be difficult to differentiate from the what you have here

[hoechenberger]

I expected the channel-specific annotation to only cover the annotated channel(s) rather than spanning all channels in one color and the annotated channel in a different (darker or lighter) version of that color.

I agree

[drammock]

I expected the channel-specific annotation to only cover the annotated channel(s) rather than spanning all channels

there is I think some sense in having channel-specific annotations be indicated somehow on the non-annotated channels (e.g. so you can see where they are in time when the already-annotated channels are offscreen). This could help you decide whether what you need is (1) a new annotation span or (2) adding a channel to an existing span.

In the office hours, I suggested using the left/right edge lines (possibly also making them dashed), and maybe (?) including the background shading in some more subtle way (thin, sparse diagonal hatching is one possibility; the same color as the annotation but with even lower alpha is another possibility).

@nmarkowitz could we see mock-ups of some or all of those approaches too?

[mscheltienne]

there is I think some sense in having channel-specific annotations be indicated somehow on the non-annotated channels (e.g. so you can see where they are in time when the already-annotated channels are offscreen)

Agree. I like the idea of using the background to denote the difference between a channel-specific and a normal annotation.

[nmarkowitz]

I expected the channel-specific annotation to only cover the annotated channel(s) rather than spanning all channels

I thought this may not be the best primarily for the reason @drammock mentioned: You wouldn't know if you were in a span that contains an annotation if the channel is offscreen. Currently this could cause making another annotation that overlaps with a channel specific annotation if you wanted to create another annotation over the same span. Eventually when the implementation of adding channels to annotations comes, showing only channels part of a specific annotation would results in many annotations being made that cover the same time span for the same reason (such as eye blink artifact).

For annotation regions/spans, the background/borders can change but something should still be in the background to indicated being in an annotation span. If an annotation continues offscreen then you may just see a colored vertical line that is hard to interpret. Adding hatching to the background in place of low alpha shading may be a solution. But even easier may just be to reduce alpha transparency of the background even more. I'll make a few more mockups to illustrate these ideas.

[nmarkowitz]

Here are some additional mockups.

This one only denotes channel-specific annotation with dashed borders. For channel that span more than the window you wouldn't know an annotation were there (without looking at the bar below). If only one of the borders of the annotation region are shown you wouldn't know in which direction the annotation spans.

---

Same as before but adds diagonal hatching to annotation region background. Looking at this I'm not a fan of it because the hatching can cause confusion/misreading when looking at the raw traces.

---

Annotation region background has a very low alpha value. In the case where an annotation region spans the entire window, maybe you can tell if it's channel specific by dash borders surrounding the window but that could be confused for boundary of the annotation.

---

Any other thoughts? I think adding something additional/different to the background of channel-specific annotations would be nice/necessary but I'm trying to think what. Any additional lines/points may confuse the eye when looking at the traces

[mscheltienne]

I would definitely keep some background, either with a different alpha and/or with a pattern/dashing added. The different alpha sounds like a good start, we could always change the background later on.

[drammock]

I think the pale alpha is a good compromise here. as @mscheltienne says we can change to a different (i.e. patterned) background later if needed.

In your mockups you have the dashed lines across the top and bottom of the window too, but I think it should be left/right edges only (this shouldn't be too hard since we already have hover effects on left/right edges, so the lines already exist, should be able to just change their styles)

[nmarkowitz]

@drammock think I should start trying to write a piece of code to start this? And I agree about the dashed lines only being on the left and right. I was just lazy in my mockup.

[drammock]

think I should start trying to write a piece of code to start this?

yeah go for it