fork() being used for multiprocessing crashes solving process with cplex

[thomgeo]

Version Checks (indicate both or one)

• [x] I have confirmed this bug exists on the lastest release of PyPSA-Eur.

• [x] I have confirmed this bug exists on the current master branch of PyPSA-Eur.

Issue Description

Dear all,

I have run into issues with PyPSA, getting the following error messages when solving a large pypsa network with cplex:

This process (pid=2637151) is multi-threaded, use of fork() may lead to deadlocks in the child.

Afterwards, the script crashes, which I assume is due to fork being used instead of spawn.

When interrupting the process early, and continuing to work on the networks, I further get this error message:

ERROR! Session/line number was not unique in database. History logging moved to new session 2301

which appears to be a result of working on a process after forks have been created.

Do you have any idea, what could cause this, and how this could be addressed? With Highs, I did not encounter these issues.

Many thanks in advance!

Reproducible Example

import pypsa

n = pypsa.Network("resources/networks/base_s_adm_2190seg__2040.nc")

n.optimize(solver_name="cplex")

Expected Behavior

network is solved and delivers results

Installed Versions

- tqdm=4.67.1=pyhd8ed1ab_1 - traitlets=5.14.3=pyhd8ed1ab_1 - typing-extensions=4.15.0=h396c80c_0 - typing_extensions=4.15.0=pyhcf101f3_0 - typing_utils=0.1.0=pyhd8ed1ab_1 - tzdata=2025b=h78e105d_0 - ukkonen=1.0.1=py312hd9148b4_6 - unicodedata2=17.0.0=py312h4c3975b_0 - unidecode=1.4.0=pyhcf101f3_1 - uri-template=1.3.0=pyhd8ed1ab_1 - uriparser=0.9.8=hac33072_0 - urllib3=2.5.0=pyhd8ed1ab_0 - validators=0.35.0=pyhd8ed1ab_0 - virtualenv=20.35.4=pyhd8ed1ab_0 - wayland=1.24.0=hd6090a7_1 - wcwidth=0.2.14=pyhd8ed1ab_0 - webcolors=25.10.0=pyhd8ed1ab_0 - webencodings=0.5.1=pyhd8ed1ab_3 - websocket-client=1.9.0=pyhd8ed1ab_0 - wheel=0.45.1=pyhd8ed1ab_1 - widgetsnbextension=3.6.10=pyhd8ed1ab_0 - wrapt=1.17.3=py312h4c3975b_1 - xarray=2025.6.1=pyhd8ed1ab_1 - xcb-util=0.4.1=h4f16b4b_2 - xcb-util-cursor=0.1.5=hb9d3cd8_0 - xcb-util-image=0.4.0=hb711507_2 - xcb-util-keysyms=0.4.1=hb711507_0 - xcb-util-renderutil=0.3.10=hb711507_0 - xcb-util-wm=0.4.2=hb711507_0 - xerces-c=3.2.5=h988505b_2 - xkeyboard-config=2.46=hb03c661_0 - xlrd=2.0.2=pyhd8ed1ab_0 - xorg-libice=1.1.2=hb9d3cd8_0 - xorg-libsm=1.2.6=he73a12e_0 - xorg-libx11=1.8.12=h4f16b4b_0 - xorg-libxau=1.0.12=hb9d3cd8_0 - xorg-libxcomposite=0.4.6=hb9d3cd8_2 - xorg-libxcursor=1.2.3=hb9d3cd8_0 - xorg-libxdamage=1.1.6=hb9d3cd8_0 - xorg-libxdmcp=1.1.5=hb9d3cd8_0 - xorg-libxext=1.3.6=hb9d3cd8_0 - xorg-libxfixes=6.0.2=hb03c661_0 - xorg-libxi=1.8.2=hb9d3cd8_0 - xorg-libxinerama=1.1.5=h5888daf_1 - xorg-libxrandr=1.5.4=hb9d3cd8_0 - xorg-libxrender=0.9.12=hb9d3cd8_0 - xorg-libxtst=1.2.5=hb9d3cd8_3 - xorg-libxxf86vm=1.1.6=hb9d3cd8_0 - xorg-xorgproto=2024.1=hb9d3cd8_1 - xyzservices=2025.10.0=pyhd8ed1ab_0 - xz=5.8.1=hbcc6ac9_2 - xz-gpl-tools=5.8.1=hbcc6ac9_2 - xz-tools=5.8.1=hb9d3cd8_2 - yaml=0.2.5=h280c20c_3 - yarl=1.22.0=py312h8a5da7c_0 - yte=1.8.1=pyha770c72_0 - zeromq=4.3.5=h387f397_9 - zict=3.0.0=pyhd8ed1ab_1 - zipp=3.23.0=pyhd8ed1ab_0 - zlib=1.3.1=hb9d3cd8_2 - zlib-ng=2.2.5=hde8ca8f_0 - zstandard=0.25.0=py312h5253ce2_0 - zstd=1.5.7=hb8e6e7a_2

[coroa]

The message is from the memory logger that is sitting in a separate forked process. I guess we could move that into a spawned process, feel free to propose a PR.

Note that we never had any actual issues with it though. I am using CPLEX 9.4 without any issues with it. It is more likely you are running out of memory and your process is being killed by the kernel. Look at the memory log file to judge how likely that is.