[GSoC] Parallelisation of AnalysisBase with multiprocessing and dask

[marinegor]

Fixes #4158 ~~Also fixes # 4259 as a check in AnalysisBase.run()~~ no it doesn't, I'll do another PR that does that.

~~Related to #4158, does not help f-i-x-i-n-g (cudos to github bot) the issue per se but paves the way towards that.~~

Changes made in this Pull Request:

• several methods added analysis.base.AnalysisBase, implementing backend configuration, splitting of the frames for analysis, computation, and results aggregation
• module analysis.backends introduces BackendBase class, as well as built-in backends BackendMultiprocessing, BackendSerial and BackendDask, implementing the apply method for computations using various backends
• module analysis.results introduces ResultsGroup class that allows for merging of multiple uniform Results objects from the same module, given appropriate aggregation functions

PR Checklist

• [x] Tests?
• [x] Docs?
• [x] CHANGELOG updated?
• [x] Issue raised/referenced?

---

:books: Documentation preview :books:: https://mdanalysis--4162.org.readthedocs.build/en/4162/

[github-actions[bot]]

Linter Bot Results:

Hi @marinegor! Thanks for making this PR. We linted your code and found the following:

Some issues were found with the formatting of your code.

Code Location	Outcome
main package	⚠️ Possible failure
testsuite	⚠️ Possible failure

Please have a look at the darker-main-code and darker-test-code steps here for more details: https://github.com/MDAnalysis/mdanalysis/actions/runs/10422317333/job/28866526863

---

Please note: The black linter is purely informational, you can safely ignore these outcomes if there are no flake8 failures!

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 97.10744% with 7 lines in your changes missing coverage. Please review.

Project coverage is 93.62%. Comparing base (d16b8d4) to head (1dc4613). Report is 2 commits behind head on develop.

Files	Patch %	Lines
package/MDAnalysis/analysis/base.py	96.52%	2 Missing and 2 partials :warning:
package/MDAnalysis/analysis/results.py	95.71%	2 Missing and 1 partial :warning:

Additional details and impacted files

@@            Coverage Diff            @@
##           develop    #4162    +/-   ##
=========================================
  Coverage    93.61%   93.62%            
=========================================
  Files          171      173     +2     
  Lines        21254    21419   +165     
  Branches      3937     3978    +41     
=========================================
+ Hits         19897    20053   +156     
- Misses         898      903     +5     
- Partials       459      463     +4     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[RMeli]

Great to see things are moving @marinegor!

I'll have a proper look at this PR when back from holidays, but I just wanted to point out that GitHub is not very smart, so "Does not fix #4162" will actually automatically close #4162 when this is merged. "Related to #4162" or something similar (with no "fix" before the PR number) would avoid this issue.

[orbeckst]

@yuxuanzhuang any initial comments so that @marinegor can move forward?

[marinegor]

Thanks! My initial thoughts were that it's handled by the CI system itself, via catching a particular type of exception that is being raised (ImportError, namely). But now I realise it has nothing to do with the CI system but with tests themselves.

[orbeckst]

As discussed today: Please break the changes to other Analysis classes except RMSD into separate individual PRs. Focus this PR on

• changes to AnalysisBase (the new parallel infrastructure)
• changes to tests
• one example: RMSD

[orbeckst]

The other thing to think about is how to aggregate results from multiple workers in a clean manner. At the meeting we favored the idea to enhance the Results data structure to annotate each item with an aggregator function. (We can define default aggregators by data type and then that would resemble the current code and potentially make it further backwards compatible.)

[IAlibay]

@marinegor we are changing the terms under which future contributions to MDAnalysis are contributed. Could you please confirm that you agree to releasing this code under the terms of the LGPLv2.1 and that your contribution also adheres to the developer certificate of origion?

[marinegor]

@IAlibay I do confirm that I agree with the terms, as well as that my contribution adheres to the certificate of origion👍

[RMeli]

@marinegor I noticed going through some of the comments that there are some "resolved conversations" that do not have "outdated" code. Please make sure you push the relevant changes before marking something as "resolved", because it facilitate reviews.

[orbeckst]

available_backends is a crucial part of the new API. I think my biggest conceptual problem is that adding a new backend (e.g., "mpi") will require changing every single class in MDAnalysis and elsewhere. This is impractical and defeats the idea of having a central AnalysisBase class that allows propagating of updates to user code. If I wrote an analysis class based on AnalysisBase then I should be able to profit from any changes.

@p-j-smith suggested in https://github.com/MDAnalysis/mdanalysis/issues/4158#issuecomment-1694694120 to mark AnalysisBase as either "serial" (=AnalysisBase) or "parallelizable" (=ParallelAnalysisBase). I don't know if using two classes is the right approach here but the idea makes some sense to me: I should really only care about if the underlying algorithm works in parallel or not.

• If parallelizable == True then I can worry about backends, but I should be able to leave it to ParallelExecutor to tell me if I can use a backend or not — it may fail with a RuntimeError if a specific backend doesn't work but I'd be ok with that.
• If parallelizable == False then that's automatically equivalent to backend == "local" and there's nothing else to do.

@marinegor and everybody else, what are your thoughts?

[marinegor]

Note to self: latest PR failed with timeout, otherwise all tests are fine.

[pep8speaks]

Hello @marinegor! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

• In the file package/MDAnalysis/analysis/backends.py:

Line 95:80: E501 line too long (81 > 79 characters) Line 122:80: E501 line too long (85 > 79 characters) Line 123:80: E501 line too long (81 > 79 characters) Line 161:80: E501 line too long (81 > 79 characters) Line 236:80: E501 line too long (98 > 79 characters) Line 272:80: E501 line too long (82 > 79 characters) Line 317:80: E501 line too long (81 > 79 characters)

• In the file package/MDAnalysis/analysis/base.py:

Line 111:80: E501 line too long (80 > 79 characters) Line 125:1: W293 blank line contains whitespace Line 127:80: E501 line too long (81 > 79 characters) Line 162:80: E501 line too long (85 > 79 characters) Line 279:80: E501 line too long (80 > 79 characters) Line 284:80: E501 line too long (83 > 79 characters) Line 291:68: W291 trailing whitespace Line 292:68: W291 trailing whitespace Line 293:64: W291 trailing whitespace Line 294:50: W291 trailing whitespace Line 304:1: W293 blank line contains whitespace Line 305:80: E501 line too long (86 > 79 characters) Line 307:80: E501 line too long (80 > 79 characters) Line 308:80: E501 line too long (80 > 79 characters) Line 310:1: W293 blank line contains whitespace Line 342:80: E501 line too long (81 > 79 characters) Line 350:80: E501 line too long (84 > 79 characters) Line 356:80: E501 line too long (83 > 79 characters) Line 365:80: E501 line too long (82 > 79 characters) Line 368:80: E501 line too long (81 > 79 characters) Line 374:80: E501 line too long (82 > 79 characters) Line 391:80: E501 line too long (87 > 79 characters) Line 420:1: W293 blank line contains whitespace Line 424:1: W293 blank line contains whitespace Line 445:80: E501 line too long (80 > 79 characters) Line 446:80: E501 line too long (85 > 79 characters) Line 453:55: W291 trailing whitespace Line 454:1: W293 blank line contains whitespace Line 457:80: E501 line too long (80 > 79 characters) Line 475:80: E501 line too long (80 > 79 characters) Line 505:80: E501 line too long (82 > 79 characters) Line 548:80: E501 line too long (80 > 79 characters) Line 567:80: E501 line too long (87 > 79 characters) Line 579:80: E501 line too long (83 > 79 characters) Line 611:80: E501 line too long (82 > 79 characters) Line 619:80: E501 line too long (85 > 79 characters) Line 625:80: E501 line too long (81 > 79 characters) Line 654:80: E501 line too long (84 > 79 characters) Line 665:80: E501 line too long (81 > 79 characters) Line 683:80: E501 line too long (81 > 79 characters) Line 716:80: E501 line too long (80 > 79 characters) Line 763:80: E501 line too long (80 > 79 characters) Line 769:80: E501 line too long (85 > 79 characters) Line 770:48: W291 trailing whitespace Line 771:40: W291 trailing whitespace Line 772:13: E129 visually indented line with same indent as next logical line Line 773:80: E501 line too long (83 > 79 characters) Line 780:80: E501 line too long (85 > 79 characters) Line 894:80: E501 line too long (87 > 79 characters) Line 934:80: E501 line too long (80 > 79 characters) Line 995:80: E501 line too long (85 > 79 characters)

• In the file package/MDAnalysis/analysis/results.py:

Line 1:80: E501 line too long (80 > 79 characters) Line 13:66: W291 trailing whitespace Line 25:75: W291 trailing whitespace Line 33:1: W293 blank line contains whitespace Line 68:80: E501 line too long (105 > 79 characters) Line 103:80: E501 line too long (80 > 79 characters) Line 128:80: E501 line too long (88 > 79 characters) Line 134:80: E501 line too long (88 > 79 characters) Line 147:80: E501 line too long (89 > 79 characters) Line 148:80: E501 line too long (89 > 79 characters) Line 149:80: E501 line too long (87 > 79 characters) Line 150:80: E501 line too long (88 > 79 characters) Line 151:80: E501 line too long (88 > 79 characters) Line 169:1: W293 blank line contains whitespace Line 177:80: E501 line too long (83 > 79 characters) Line 185:80: E501 line too long (97 > 79 characters) Line 186:66: W291 trailing whitespace Line 188:79: W291 trailing whitespace Line 189:80: E501 line too long (81 > 79 characters) Line 190:80: E501 line too long (84 > 79 characters) Line 211:80: E501 line too long (94 > 79 characters) Line 216:1: W293 blank line contains whitespace

• In the file package/MDAnalysis/analysis/rms.py:

Line 362:80: E501 line too long (81 > 79 characters)

• In the file package/MDAnalysis/lib/util.py:

Line 2612:1: W293 blank line contains whitespace Line 2613:1: W293 blank line contains whitespace Line 2616:60: W291 trailing whitespace

• In the file testsuite/MDAnalysisTests/analysis/test_base.py:

Line 45:80: E501 line too long (81 > 79 characters) Line 63:80: E501 line too long (84 > 79 characters) Line 96:1: E302 expected 2 blank lines, found 1 Line 102:1: E302 expected 2 blank lines, found 1 Line 105:1: E302 expected 2 blank lines, found 1 Line 111:1: E302 expected 2 blank lines, found 1 Line 115:1: E302 expected 2 blank lines, found 1 Line 119:1: E302 expected 2 blank lines, found 1 Line 124:1: E302 expected 2 blank lines, found 1 Line 137:80: E501 line too long (85 > 79 characters) Line 139:1: E302 expected 2 blank lines, found 1 Line 143:80: E501 line too long (96 > 79 characters) Line 146:80: E501 line too long (116 > 79 characters) Line 148:1: E302 expected 2 blank lines, found 1 Line 155:80: E501 line too long (108 > 79 characters) Line 157:1: E302 expected 2 blank lines, found 1 Line 162:80: E501 line too long (83 > 79 characters) Line 165:80: E501 line too long (112 > 79 characters) Line 168:80: E501 line too long (86 > 79 characters) Line 170:1: E302 expected 2 blank lines, found 1 Line 180:80: E501 line too long (116 > 79 characters) Line 182:80: E501 line too long (107 > 79 characters) Line 184:1: E302 expected 2 blank lines, found 1 Line 191:80: E501 line too long (93 > 79 characters) Line 192:80: E501 line too long (87 > 79 characters) Line 261:1: E302 expected 2 blank lines, found 1 Line 262:47: W291 trailing whitespace Line 264:50: W291 trailing whitespace Line 275:1: E302 expected 2 blank lines, found 1 Line 286:80: E501 line too long (88 > 79 characters) Line 292:80: E501 line too long (89 > 79 characters) Line 349:80: E501 line too long (80 > 79 characters) Line 401:80: E501 line too long (90 > 79 characters) Line 402:80: E501 line too long (100 > 79 characters) Line 403:80: E501 line too long (87 > 79 characters) Line 459:80: E501 line too long (87 > 79 characters)

• In the file testsuite/MDAnalysisTests/analysis/test_pca.py:

Line 139:43: E231 missing whitespace after ','

• In the file testsuite/MDAnalysisTests/analysis/test_results.py:

Line 160:80: E501 line too long (82 > 79 characters) Line 173:80: E501 line too long (114 > 79 characters)

• In the file testsuite/MDAnalysisTests/analysis/test_rms.py:

Line 208:80: E501 line too long (81 > 79 characters) Line 218:80: E501 line too long (81 > 79 characters) Line 220:80: E501 line too long (81 > 79 characters) Line 225:80: E501 line too long (81 > 79 characters) Line 235:80: E501 line too long (87 > 79 characters) Line 242:80: E501 line too long (97 > 79 characters) Line 250:80: E501 line too long (92 > 79 characters) Line 252:80: E501 line too long (107 > 79 characters) Line 257:80: E501 line too long (84 > 79 characters) Line 260:80: E501 line too long (94 > 79 characters) Line 265:80: E501 line too long (89 > 79 characters) Line 270:80: E501 line too long (105 > 79 characters) Line 270:106: W291 trailing whitespace Line 271:80: E501 line too long (110 > 79 characters) Line 272:80: E501 line too long (97 > 79 characters) Line 272:97: E124 closing bracket does not match visual indentation Line 278:80: E501 line too long (106 > 79 characters) Line 284:80: E501 line too long (105 > 79 characters) Line 284:106: W291 trailing whitespace Line 285:80: E501 line too long (109 > 79 characters) Line 285:97: E128 continuation line under-indented for visual indent Line 286:80: E501 line too long (97 > 79 characters) Line 286:97: E124 closing bracket does not match visual indentation Line 327:80: E501 line too long (86 > 79 characters) Line 343:80: E501 line too long (85 > 79 characters) Line 362:54: E231 missing whitespace after ',' Line 362:80: E501 line too long (96 > 79 characters) Line 407:80: E501 line too long (94 > 79 characters) Line 428:80: E501 line too long (85 > 79 characters) Line 449:80: E501 line too long (80 > 79 characters)

Comment last updated at 2024-08-16 15:23:41 UTC

[marinegor]

Hi everyone, I firmly believe that I've made all the changes that are possible, including updating documentation & writing few more tests for good code coverage. Hence, I'm pinging @orbeckst @IAlibay @RMeli @p-j-smith @yuxuanzhuang again!

[RMeli]

Thanks @marinegor for all the work and for the ping. I have a lot going on the next few weeks, but I'll start having another deep look at this PR. Apologies in advance if the review comments will come in multiple batches.

[yuxuanzhuang]

Thanks! I will share my review by the end of this week :)

[orbeckst]

Very good progress @marinegor ! I am sorry, I am totally swamped and are unlikely to be able to contribute a meaningful review over the next 2 weeks. Anyone, please ping me for specific question if my opinion may be of use and I try to answer. Sorry, that's the best I can do at the moment.

[orbeckst]

@RMeli @yuxuanzhuang if my review becomes the only blocking one remaining then please ping me and I have a super-quick look and then probably approve or dismiss my review as stale.

[marinegor]

Feel like I have to ping @IAlibay @p-j-smith and @orbeckst here again for the code review

[marinegor]

I applied changes from most of the reviewers, hoping to make the work for @orbeckst and @IAlibay slightly easier.

[marinegor]

Pinging @MDAnalysis/coredevs here, since I have added a whole big page in documentation based on UGM presentation & GSOC blogpost: check it out

[marinegor]

Just wanted to re-ping @orbeckst here -- the documentation is now fixed, and contains a big tutorial entry with 2 new illustrations and content inspired by GSOC post & UGM presentation.

[RMeli]

I fixed a couple of trivial merge conflicts.

[marinegor]

@RMeli thanks a lot for the edits! I applied them everywhere (and thanks for the *not* catch!)

@MDAnalysis/coredevs is there anything else left to do here?

[orbeckst]

@IAlibay could you see if your comments were addressed when you can make some time? I am not asking for a full review, just if you're happy with what you specifically raised. My goal is to slowly start turning ❌ into ✅ .

[orbeckst]

@RMeli and @p-j-smith if you're already happy with the PR (and assuming that any other issues raised by other reviewers will be addressed) please feel free to add an approving review to the PR. If you find anything (many eyes are better than a few...) the of course raise it.

From my reading of the PR, we're getting towards merging this mammoth effort 🦣 and it would be great to have the approvals of everyone who was involved in discussions/reviews over the last year.

p.s.: There's an emoji for "mammoth" 🦣 (and "DNA" 🧬 ) but none for "protein" 🍖 ?!?

[IAlibay]

@IAlibay could you see if your comments were addressed when you can make some time? I am not asking for a full review, just if you're happy with what you specifically raised. My goal is to slowly start turning ❌ into ✅ .

Sorry for the lack of response - this is next on my big PR review todo list (last week was guessers 😅 )!

[marinegor]

Hi everyone, I fixed @yuxuanzhuang 's suggestions (test now runs, and the warning is also in place).

I also fixed a merge conflict after #4459 was introduced -- apparently, I fixed the same issue with None default value instead of {} during my PR as well.

UPD: I looked at the failing CI/CD runs, and it seems that all of them are related to either codecov: failed to properly upload..., or the timeout issue mentioned many times earlier. So I guess on my side everything is fine.

[orbeckst]

I put in my calendar for early next week "review dask PR".

[orbeckst]

I installed a Python 3.11 conda env (macOS) with this branch and I am trying out the example from the docs https://mdanalysis--4162.org.readthedocs.build/en/4162/documentation_pages/analysis_modules.html

import multiprocessing
import MDAnalysis as mda
from MDAnalysisTests.datafiles import PSF, DCD
from MDAnalysis.analysis.rms import RMSD
from MDAnalysis.analysis.align import AverageStructure

# initialize the universe
u = mda.Universe(PSF, DCD)

# calculate average structure for reference
avg = AverageStructure(mobile=u).run()
ref = avg.results.universe

# initialize RMSD run
rmsd = RMSD(u, ref, select='backbone')
rmsd.run(backend='multiprocessing', n_workers=multiprocessing.cpu_count())

I am running in a Jupyter lab notebook. When I imported MDAnalysis, I got the warning

[/Users/oliver/anaconda3/envs/mda311dask/lib/python3.11/site-packages/tqdm/auto.py:21](http://localhost:8888/Users/oliver/anaconda3/envs/mda311dask/lib/python3.11/site-packages/tqdm/auto.py#line=20): TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm

The RMSD calculation succeeds and I can plot the rmsd.results.

However, I get the following messages:

When I ran the first time:

Exception ignored in: <module 'threading' from '[/Users/oliver/anaconda3/envs/mda311dask/lib/python3.11/threading.py](http://localhost:8888/Users/oliver/anaconda3/envs/mda311dask/lib/python3.11/threading.py)'>
Exception ignored in: <module 'threading' from '[/Users/oliver/anaconda3/envs/mda311dask/lib/python3.11/threading.py](http://localhost:8888/Users/oliver/anaconda3/envs/mda311dask/lib/python3.11/threading.py)'>
Exception ignored in: <module 'threading' from '[/Users/oliver/anaconda3/envs/mda311dask/lib/python3.11/threading.py](http://localhost:8888/Users/oliver/anaconda3/envs/mda311dask/lib/python3.11/threading.py)'>
Exception ignored in: <module 'threading' from '[/Users/oliver/anaconda3/envs/mda311dask/lib/python3.11/threading.py](http://localhost:8888/Users/oliver/anaconda3/envs/mda311dask/lib/python3.11/threading.py)'>
Exception ignored in atexit callback: <bound method TMonitor.exit of <TMonitor(Thread-1, stopped daemon 123145468305408)>>
Exception ignored in atexit callback: <function exit_cacert_ctx at 0x105828f40>
Traceback (most recent call last):
Traceback (most recent call last):
  File "[/Users/oliver/anaconda3/envs/mda311dask/lib/python3.11/threading.py", line 1541](http://localhost:8888/Users/oliver/anaconda3/envs/mda311dask/lib/python3.11/threading.py#line=1540), in _shutdown
Traceback (most recent call last):
Traceback (most recent call last):
  File "[/Users/oliver/anaconda3/envs/mda311dask/lib/python3.11/site-packages/certifi/core.py", line 10](http://localhost:8888/Users/oliver/anaconda3/envs/mda311dask/lib/python3.11/site-packages/certifi/core.py#line=9), in exit_cacert_ctx
  File "[/Users/oliver/anaconda3/envs/mda311dask/lib/python3.11/site-packages/tqdm/_monitor.py", line 44](http://localhost:8888/Users/oliver/anaconda3/envs/mda311dask/lib/python3.11/site-packages/tqdm/_monitor.py#line=43), in exit
  File "[/Users/oliver/anaconda3/envs/mda311dask/lib/python3.11/threading.py", line 1541](http://localhost:8888/Users/oliver/anaconda3/envs/mda311dask/lib/python3.11/threading.py#line=1540), in _shutdown
Traceback (most recent call last):
Traceback (most recent call last):
  File "[/Users/oliver/anaconda3/envs/mda311dask/lib/python3.11/threading.py", line 1541](http://localhost:8888/Users/oliver/anaconda3/envs/mda311dask/lib/python3.11/threading.py#line=1540), in _shutdown
  File "[/Users/oliver/anaconda3/envs/mda311dask/lib/python3.11/threading.py", line 1541](http://localhost:8888/Users/oliver/anaconda3/envs/mda311dask/lib/python3.11/threading.py#line=1540), in _shutdown
    self.join()
  File "[/Users/oliver/anaconda3/envs/mda311dask/lib/python3.11/threading.py", line 1119](http://localhost:8888/Users/oliver/anaconda3/envs/mda311dask/lib/python3.11/threading.py#line=1118), in join
    def exit_cacert_ctx() -> None:
    
  File "[/Users/oliver/anaconda3/envs/mda311dask/lib/python3.11/site-packages/gsd/__init__.py", line 25](http://localhost:8888/Users/oliver/anaconda3/envs/mda311dask/lib/python3.11/site-packages/gsd/__init__.py#line=24), in <lambda>
    self._wait_for_tstate_lock()
  File "[/Users/oliver/anaconda3/envs/mda311dask/lib/python3.11/threading.py", line 1139](http://localhost:8888/Users/oliver/anaconda3/envs/mda311dask/lib/python3.11/threading.py#line=1138), in _wait_for_tstate_lock
    signal.signal(signal.SIGTERM, lambda n, f: sys.exit(1))
                                               ^^^^^^^^^^^
SystemExit: 1
    if lock.acquire(block, timeout):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[/Users/oliver/anaconda3/envs/mda311dask/lib/python3.11/site-packages/gsd/__init__.py", line 25](http://localhost:8888/Users/oliver/anaconda3/envs/mda311dask/lib/python3.11/site-packages/gsd/__init__.py#line=24), in <lambda>
    def _shutdown():
    def _shutdown():
    
    
  File "[/Users/oliver/anaconda3/envs/mda311dask/lib/python3.11/site-packages/gsd/__init__.py", line 25](http://localhost:8888/Users/oliver/anaconda3/envs/mda311dask/lib/python3.11/site-packages/gsd/__init__.py#line=24), in <lambda>
  File "[/Users/oliver/anaconda3/envs/mda311dask/lib/python3.11/site-packages/gsd/__init__.py", line 25](http://localhost:8888/Users/oliver/anaconda3/envs/mda311dask/lib/python3.11/site-packages/gsd/__init__.py#line=24), in <lambda>
    def _shutdown():
    
    def _shutdown():
  File "[/Users/oliver/anaconda3/envs/mda311dask/lib/python3.11/site-packages/gsd/__init__.py", line 25](http://localhost:8888/Users/oliver/anaconda3/envs/mda311dask/lib/python3.11/site-packages/gsd/__init__.py#line=24), in <lambda>
    
  File "[/Users/oliver/anaconda3/envs/mda311dask/lib/python3.11/site-packages/gsd/__init__.py", line 25](http://localhost:8888/Users/oliver/anaconda3/envs/mda311dask/lib/python3.11/site-packages/gsd/__init__.py#line=24), in <lambda>
    signal.signal(signal.SIGTERM, lambda n, f: sys.exit(1))
    signal.signal(signal.SIGTERM, lambda n, f: sys.exit(1))
                                               ^^^^^^^^^^^
SystemExit: 1
    signal.signal(signal.SIGTERM, lambda n, f: sys.exit(1))
    signal.signal(signal.SIGTERM, lambda n, f: sys.exit(1))
                                               ^^^^^^^^^^^
                                               ^^^^^^^^^^^
SystemExit: 1
SystemExit: 1
    signal.signal(signal.SIGTERM, lambda n, f: sys.exit(1))
                                               ^^^^^^^^^^^
SystemExit: 1
                                               ^^^^^^^^^^^
SystemExit: 1

When I ran a second time the same command

rmsd.run(backend='multiprocessing', n_workers=multiprocessing.cpu_count())

no such messages appeared.

When I run with a different number of workers

rmsd.run(backend='multiprocessing', n_workers=2)

the following shorter messages appear:

Exception ignored in atexit callback: <function shutdown at 0x1049751c0>
Traceback (most recent call last):
  File "[/Users/oliver/anaconda3/envs/mda311dask/lib/python3.11/logging/__init__.py", line 2181](http://localhost:8888/Users/oliver/anaconda3/envs/mda311dask/lib/python3.11/logging/__init__.py#line=2180), in shutdown
    h = wr()
        ^^^^
  File "[/Users/oliver/anaconda3/envs/mda311dask/lib/python3.11/site-packages/gsd/__init__.py", line 25](http://localhost:8888/Users/oliver/anaconda3/envs/mda311dask/lib/python3.11/site-packages/gsd/__init__.py#line=24), in <lambda>
    signal.signal(signal.SIGTERM, lambda n, f: sys.exit(1))
                                               ^^^^^^^^^^^
SystemExit: 1

Run the same again:

Exception ignored in atexit callback: <function exit_cacert_ctx at 0x1149ccf40>
Traceback (most recent call last):
  File "[/Users/oliver/anaconda3/envs/mda311dask/lib/python3.11/site-packages/certifi/core.py", line 10](http://localhost:8888/Users/oliver/anaconda3/envs/mda311dask/lib/python3.11/site-packages/certifi/core.py#line=9), in exit_cacert_ctx
Exception ignored in atexit callback: <function exit_cacert_ctx at 0x11357cf40>
Traceback (most recent call last):
  File "[/Users/oliver/anaconda3/envs/mda311dask/lib/python3.11/site-packages/certifi/core.py", line 10](http://localhost:8888/Users/oliver/anaconda3/envs/mda311dask/lib/python3.11/site-packages/certifi/core.py#line=9), in exit_cacert_ctx
    def exit_cacert_ctx() -> None:
    def exit_cacert_ctx() -> None:
    
  File "[/Users/oliver/anaconda3/envs/mda311dask/lib/python3.11/site-packages/gsd/__init__.py", line 25](http://localhost:8888/Users/oliver/anaconda3/envs/mda311dask/lib/python3.11/site-packages/gsd/__init__.py#line=24), in <lambda>
    
  File "[/Users/oliver/anaconda3/envs/mda311dask/lib/python3.11/site-packages/gsd/__init__.py", line 25](http://localhost:8888/Users/oliver/anaconda3/envs/mda311dask/lib/python3.11/site-packages/gsd/__init__.py#line=24), in <lambda>
    signal.signal(signal.SIGTERM, lambda n, f: sys.exit(1))
                                               ^^^^^^^^^^^
SystemExit: 1
    signal.signal(signal.SIGTERM, lambda n, f: sys.exit(1))
                                               ^^^^^^^^^^^
SystemExit: 1

Run again

Exception ignored in atexit callback: <function exit_cacert_ctx at 0x10e658f40>
Traceback (most recent call last):
  File "[/Users/oliver/anaconda3/envs/mda311dask/lib/python3.11/site-packages/certifi/core.py", line 10](http://localhost:8888/Users/oliver/anaconda3/envs/mda311dask/lib/python3.11/site-packages/certifi/core.py#line=9), in exit_cacert_ctx
    def exit_cacert_ctx() -> None:
    
  File "[/Users/oliver/anaconda3/envs/mda311dask/lib/python3.11/site-packages/gsd/__init__.py", line 25](http://localhost:8888/Users/oliver/anaconda3/envs/mda311dask/lib/python3.11/site-packages/gsd/__init__.py#line=24), in <lambda>
    signal.signal(signal.SIGTERM, lambda n, f: sys.exit(1))
                                               ^^^^^^^^^^^
SystemExit: 1

I am fairly confused what this is about. Can someone else reproduce?

But I know that I don't want our users to see these kind of messages.

Note that running in a script

#!/usr/bin/env python
# -*- coding: utf-8 -*-
# https://github.com/MDAnalysis/mdanalysis/pull/4162

import multiprocessing
import MDAnalysis as mda
from MDAnalysisTests.datafiles import PSF, DCD
from MDAnalysis.analysis.rms import RMSD
from MDAnalysis.analysis.align import AverageStructure

if __name__ == "__main__":
    # must all be in __main__, otherwise multiprocessing loops indefinitely

    # initialize the universe
    u = mda.Universe(PSF, DCD)

    # calculate average structure for reference
    avg = AverageStructure(mobile=u).run()
    ref = avg.results.universe

    # initialize RMSD run
    rmsd = RMSD(u, ref, select='backbone')
    rmsd.run(backend='multiprocessing', n_workers=multiprocessing.cpu_count())

does not produce the above messages, so it may be specific for jupyter lab.