ENH: Direct function call for pipeline in Python script

[scholzri]

First of all, thank you for creating this amazing pipeline. It's been incredibly helpful in my work.

I'm currently integrating the mne-bids-pipeline into a larger Python script that handles data conversion and trigger recoding. I'd like to know if there is a way to execute specific pipeline steps directly from within my script.

Currently, I'm using the command-line interface like this:

mne_bids_pipeline config.py --steps=preprocessing

Is there a function I can import and call in my Python script that achieves the same result? Something like:

from mne_bids_pipeline import run_pipeline
run_pipeline('config.py', steps=['preprocessing'])

If not, is the recommended approach to use os.system() or subprocess.run() to execute the command-line interface? Any guidance on the best practice for integrating mne-bids-pipeline steps into custom Python scripts would be greatly appreciated. Thank you!

[welcome[bot]]

Hello! 👋 Thanks for opening your first issue here! ❤️ We will try to get back to you soon. 🚴🏽‍♂️

[larsoner]

We currently don't really have a public API for this sort of thing. Looking at what the command-line mne_bids_pipeline actually invokes:

https://github.com/mne-tools/mne-bids-pipeline/blob/cce897d3ac3324a6ab5fbd5f7333e1df3c61f919/mne_bids_pipeline/_main.py#L17

The two parts I think you need are

1. importing and validating a config file
2. running a single step using the given config variable

Once you have (2) you trivially have support for multiple steps. And we are very close to exposing (2) publicly since I think it's currently available at mne_bids_pipeline.steps.preprocessing._01_whatever.main(...). We could make (1) public probably -- there is currently a _import_config that we could give a nice name like import_user_config(path) or similar.

I think this would work for your use case and not require a ton of extra work at our end, does that sound right? If so then we could start with a PR to add these steps and test it in the GHA CIs without too much extra work now that @drammock has written some nice unit test shims!