Export epoch issue

[arnodelorme]

File available at https://drive.google.com/file/d/1RY5LEM_mpOdF5Ka0nTe6JGMvo_kDk1M5/view?usp=sharing

import mne

from mne.datasets import sample
filename = 'sub-001_task-P300_run-2_eeg.set';
raw = mne.io.read_raw_eeglab(filename)
events_from_annot, event_dict = mne.events_from_annotations(raw)
epochs_all = mne.Epochs(raw, events_from_annot, tmin=-0.3, tmax=0.7, event_id=event_dict, preload=True)
epochs_all.export('test.set', overwrite=True)

The exported file appears corrupted on MATLAB as the number of epoch in the epoch structure does not match the actual number of epochs. In fact EEGLAB generate a corrupted file error. You can simply set EEG.epoch to empty if necessary as EEGLAB can regenerate automatically from the events.

TMP = load('-mat', 'test.set')

TMP = 

  struct with fields:

          data: [79×257×861 double]
       setname: '/System/Volumes/Data/data/matlab/ds003061_process/MNE/test.set'
        nbchan: 79
          pnts: 257
        trials: 861
         srate: 256
          xmin: -0.3008
          xmax: 0.6992
           ref: 'common'
      chanlocs: [1×79 struct]
         event: [1×861 struct]
         epoch: [1×749 struct]
       icawinv: []
     icasphere: []
    icaweights: []

[arnodelorme]

Platform:         macOS-10.16-x86_64-i386-64bit
Python:           3.8.5 (default, Sep  4 2020, 02:22:02)  [Clang 10.0.0 ]
Executable:       /Users/arno/miniconda3/envs/p38env/bin/python
CPU:              i386: 16 cores
Memory:           Unavailable (requires "psutil" package)
mne:              1.1.0
numpy:            1.19.2 {blas=mkl_rt, lapack=mkl_rt}
scipy:            1.4.1
matplotlib:       3.3.3 {backend=module://ipykernel.pylab.backend_inline}

sklearn:          0.24.1
numba:            Not found
nibabel:          Not found
nilearn:          Not found
dipy:             Not found
cupy:             Not found
pandas:           1.2.1
pyvista:          Not found
pyvistaqt:        Not found
ipyvtklink:       Not found
vtk:              Not found
qtpy:             Not found
ipympl:           Not found
pyqtgraph:        Not found
pooch:            v1.6.0

mne_bids:         0.10
mne_nirs:         Not found
mne_features:     Not found
mne_qt_browser:   Not found
mne_connectivity: Not found
mne_icalabel:     Not found

[agramfort]

ping @jackz314 @cbrnr

[jackz314]

@arnodelorme could you try this again with the main branch of eeglabio? I had this issue earlier as well and I believe the latest version should fix it.

Also, the epoch struct needs to be exported since mne uses it to read events/epochs, but perhaps I could add an option to not export this field if the user doesn't care about loading the file with mne again.

[cbrnr]

could you try this again with the main branch of eeglabio? I had this issue earlier as well and I believe the latest version should fix it.

I tried it with eeglabio-0.0.2.post1. I don't get an error/warning when exporting, but loading the file fails:

In [4]: raw_new = mne.read_epochs_eeglab("/Users/clemens/Desktop/test.set")
<ipython-input-4-ecb3ec98f7aa>:1: RuntimeWarning: At least one epoch has multiple events. Only the latency of the first event will be retained.
  raw_new = mne.read_epochs_eeglab("/Users/clemens/Desktop/test.set")
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
Input In [4], in <cell line: 1>()
----> 1 raw_new = mne.read_epochs_eeglab("/Users/clemens/Desktop/test.set")

File ~/Projects/mne-python/mne/io/eeglab/eeglab.py:317, in read_epochs_eeglab(input_fname, events, event_id, eog, verbose, uint16_codec)
    265 @fill_doc
    266 def read_epochs_eeglab(input_fname, events=None, event_id=None,
    267                        eog=(), verbose=None, uint16_codec=None):
    268     r"""Reader function for EEGLAB epochs files.
    269 
    270     Parameters
   (...)
    315     .. versionadded:: 0.11.0
    316     """
--> 317     epochs = EpochsEEGLAB(input_fname=input_fname, events=events, eog=eog,
    318                           event_id=event_id, verbose=verbose,
    319                           uint16_codec=uint16_codec)
    320     return epochs

File <decorator-gen-278>:12, in __init__(self, input_fname, events, event_id, tmin, baseline, reject, flat, reject_tmin, reject_tmax, eog, verbose, uint16_codec)

File ~/Projects/mne-python/mne/io/eeglab/eeglab.py:545, in EpochsEEGLAB.__init__(self, input_fname, events, event_id, tmin, baseline, reject, flat, reject_tmin, reject_tmax, eog, verbose, uint16_codec)
    542 if idx == 0:
    543     prev_stim = 0
    544 elif (idx > 0 and
--> 545         event_latencies[idx] - event_latencies[idx - 1] == 1):
    546     prev_stim = event_id[event_name[idx - 1]]
    547 events[idx, 0] = event_latencies[idx]

IndexError: list index out of range

Also, the epoch struct needs to be exported since mne uses it to read events/epochs, but perhaps I could add an option to not export this field if the user doesn't care about loading the file with mne again.

I wouldn't do that. It doesn't hurt to include it.

[jackz314]

I tried it with eeglabio-0.0.2.post1. I don't get an error/warning when exporting, but loading the file fails

I meant from the main branch directly, not the latest release, I've been holding off on the new release until I've tested things more thoroughly.

[cbrnr]

Got it. Yes, this works without any errors, I only get these two warnings(?) when exporting:

/Users/clemens/Projects/mne-python/.direnv/python-3.10.6/lib/python3.10/site-packages/eeglabio/epochs.py:135 EEGLABIO: WARNING: Events doesn't meet the requirement of at least one event per epoch, adding dummy events
<ipython-input-1-edcbccf93868>:8: RuntimeWarning: At least one epoch has multiple events. Only the latency of the first event will be retained.

Importing works fine, there are 861 epochs:

<EpochsEEGLAB |  861 events (all good), -0.300781 - 0.699219 sec, baseline off, ~133.5 MB, data loaded,
 'dummy': 111
 'standard/standard': 486
 'noise_with_reponse/noise_with_reponse/response/response': 1
 'noise/noise': 111
 'oddball_with_reponse/oddball_with_reponse/response/response': 71
 'oddball_with_reponse/oddball_with_reponse': 39
 'response/response/standard/standard': 39
 'oddball_with_reponse/oddball_with_reponse/response': 1
 'response/standard/standard': 1
 'standard': 1>

Is this correct @arnodelorme?

[arnodelorme]

How do I try with the main branch? Can I uninstall with pip then clone the repo? Would be great to be able to make push requests if needed.

[drammock]

pip install git+https://github.com/jackz314/eeglabio will install current main branch. If you want to make pull requests, then I'd first fork eeglabio, then

git clone https://github.com/arnodelorme/eeglabio.git
cd eeglabio
git remote add upstream https://github.com/jackz314/eeglabio.git
pip install -e .

The -e flag does an "editable" install: python will load eeglabio from your local clone including any changes you have made locally.

[jackz314]

FYI I finished tests on my end and released the latest version as v0.0.2-2.

[larsoner]

I think this has been fixed by eeglabio, but feel free to reopen if it still occurs after updating!