Index bug when annotating segments in data browser

[richardkoehler]

Description of the problem

Description

When annotating bad segments in the QtBrowser, we noticed that if the bad segment includes the very end of the recording, the last sample of the recording is not discarded, when running: raw, times = raw.get_data(reject_by_annotation="omit", return_times=True)

On the rest of the bad segment it works fine, including the second to last sample (see code below) - it is simply the last sample that does not seem to be discarded. You can observe when plotting the raw data afterwards:

and when inspecting the last two samples of the times:

I included a code sample, but to reproduce the bug yourself, you have to manually shift the annotation in the raw browser so that it slightly "depasses" the end of the recording.

Proposal

I suspect that this might be related to indexing in a half-open interval (i.e. the last sample of the annotated segment in the data browser is not included in the raw.annotation, as in numpy.arange etc.). If this is the case, it would not be possible to annotate the last sample of the recording, which I would consider a bug worth fixing, even if it is obviously an edge case.

Discussion

Let me know if you think this would be desirable to fix. If you point me towards where in the source code the annotations in the data browser are translated to the annotations in the raw object, I can implement the changes myself, but unfortunately the plotting code is rather complex so I wasn't able to find the right spot in the source code myself.

Steps to reproduce

from matplotlib import pyplot as plt
from mne_bids import BIDSPath, read_raw_bids
import mne


bids_root = mne.datasets.epilepsy_ecog.data_path()

bids_path = BIDSPath(
    root=bids_root, 
    subject='pt1', 
    session='presurgery',
    task='ictal', 
    datatype='ieeg', 
    extension='.vhdr'
    )

raw: mne.io.Raw = read_raw_bids(bids_path=bids_path, verbose=False)

raw.load_data(verbose=False)

raw.set_annotations(None)

print(f"\n{raw.annotations = }")

last_time = raw.times[-1]

print(f"\nLast time point raw data: {last_time}")

data_bef, times_bef = raw.get_data(
    picks=None, start=0, stop=None, reject_by_annotation="omit", return_times=True
)

print(f"\nLast time point before: {times_bef[-1]}")
print(f"\n{len(times_bef) = }")


annotations = mne.Annotations(
    onset=[last_time - 0.999], duration=[1.1], description=["BAD_SEGMENT"],
    orig_time= raw.info['meas_date'], ch_names=[raw.info["ch_names"]]
    )

raw.set_annotations(annotations)

# Uncomment this to confirm the problem only occurs when manually shifting
# the annotation in a browser and not systematically when the annotation
# exceeds the recording duration
# raw._annotations = annotations

raw.plot(scalings="auto", block=True, verbose=True)

print(f"\n{raw.annotations = }")

data_aft, times_aft = raw.get_data(
    picks=None, start=0, stop=None, reject_by_annotation="omit", return_times=True
)

print(f"\nLast time point after: {times_aft[-1]}")
print(f"\n2nd to last time point after: {times_aft[-2]}")
print(f"\n{len(times_aft) = }")

plt.plot(times_aft[-1000:], data_aft[0,-1000:])
plt.show(block=True)

Link to data

No response

Expected results

Last time point before: 269.079

Last time point after: **268.079**

2nd to last time point after: 268.078

Actual results

Last time point before: 269.079

Last time point after: **269.079**

2nd to last time point after: 268.078

Additional information

Platform: Windows-10-10.0.22621-SP0 Python: 3.10.8 | packaged by conda-forge | (main, Nov 22 2022, 08:16:33) [MSC v.1929 64 bit (AMD64)] Executable: C:..\anaconda3\envs\mnedev\python.exe
CPU: Intel64 Family 6 Model 142 Stepping 12, GenuineIntel: 8 cores Memory: 31.8 GB

mne: 1.3.dev0 numpy: 1.23.5 {unknown linalg bindings} scipy: 1.9.3 matplotlib: 3.6.2 {backend=QtAgg}

sklearn: 1.2.0 numba: 0.56.4 nibabel: 4.0.2 nilearn: 0.9.2 dipy: 1.5.0 openmeeg: 2.5.5 cupy: Not found pandas: 1.5.2 pyvista: 0.37.0 {OpenGL 4.5.0 - Build 31.0.101.2114 via Intel(R) UHD Graphics} pyvistaqt: 0.9.0 ipyvtklink: 0.2.2 vtk: 9.2.2 qtpy: 2.3.0 {PyQt5=5.15.6} ipympl: 0.9.2 pyqtgraph: 0.13.1 pooch: v1.6.0

mne_bids: 0.12 mne_nirs: Not found mne_features: Not found mne_qt_browser: 0.4.0 mne_connectivity: Not found mne_icalabel: Not found