import of paho.mqtt.client not possible when ONLY importing in subthread

[KonstantinWaser]

Bug Description

When running a python script and trying to import paho.mqtt.client in a subthread, without first importing the library in the main thread, an exception occurs. If the library is imported before in the main thread, no problems occurr.

Reproduction

This simple code snippet:

import threading
def mqtt_thread():
    print("Starting MQTT")
    import paho.mqtt.client as mqtt
    print("MQTT imported in thread")

t = threading.Thread(target=mqtt_thread)
t.start()

executes up to the first print statement, I see the "Starting MQTT" message, then the following exception occurrs: RuntimeError: can't register atexit after shutdown

When doing the same without the subthread, there are no problems at all. Also, when first importing the package in the main thread, like here:

import threading
import paho.mqtt.client as mqtt
def mqtt_thread():
    print("Starting MQTT")
    **import paho.mqtt.client as mqtt**
    print("MQTT imported in thread")

t = threading.Thread(target=mqtt_thread)
t.start()

also works just fine.

Environment

• Python version: tested on 3.11.3 and 3.11.9
• Library version: 2.1.0
• Operating system: Windows 11 (tested on different versions)

Whole Stacktrace:

Starting MQTT Exception in thread Thread-1 (mqtt_thread): Traceback (most recent call last): File "C:\Python\311_x64\Lib\threading.py", line 1045, in _bootstrap_inner self.run() File "C:\Python\311_x64\Lib\threading.py", line 982, in run self._target(*self._args, **self._kwargs) File "c:\Users\KonstantinWaser\Desktop\pahotest.py", line 18, in mqtt_thread import paho.mqtt.client as mqtt File "C:\Python\311_x64\Lib\site-packages\paho\mqtt\client.py", line 116, in import dns.resolver File "C:\Python\311_x64\Lib\site-packages\dns\resolver.py", line 30, in import dns._ddr File "C:\Python\311_x64\Lib\site-packages\dns_ddr.py", line 12, in import dns.nameserver File "C:\Python\311_x64\Lib\site-packages\dns\nameserver.py", line 5, in import dns.asyncquery File "C:\Python\311_x64\Lib\site-packages\dns\asyncquery.py", line 38, in from dns.query import ( File "C:\Python\311_x64\Lib\site-packages\dns\query.py", line 64, in import httpcore.backends.sync File "C:\Python\311_x64\Lib\site-packages\httpcore_init.py", line 1, in from ._api import request, stream File "C:\Python\311_x64\Lib\site-packages\httpcore_api.py", line 5, in from .sync.connection_pool import ConnectionPool File "C:\Python\311_x64\Lib\site-packages\httpcore_sync_init.py", line 1, in from .connection import HTTPConnection File "C:\Python\311_x64\Lib\site-packages\httpcore_sync\connection.py", line 12, in from ..synchronization import Lock File "C:\Python\311_x64\Lib\site-packages\httpcore_synchronization.py", line 16, in import anyio File "C:\Python\311_x64\Lib\site-packages\anyio_init.py", line 21, in from ._core.fileio import AsyncFile as AsyncFile File "C:\Python\311_x64\Lib\site-packages\anyio_core_fileio.py", line 21, in from .. import to_thread File "C:\Python\311_x64\Lib\site-packages\anyio\to_thread.py", line 9, in from .abc import CapacityLimiter File "C:\Python\311_x64\Lib\site-packages\anyio\abc_init.py", line 50, in from ..from_thread import BlockingPortal as BlockingPortal File "C:\Python\311_x64\Lib\site-packages\anyio\from_thread.py", line 6, in from concurrent.futures import FIRST_COMPLETED, Future, ThreadPoolExecutor, wait File "", line 1229, in handle_fromlist File "C:\Python\311_x64\Lib\concurrent\futures_init.py", line 49, in getattr from .thread import ThreadPoolExecutor as te File "C:\Python\311_x64\Lib\concurrent\futures\thread.py", line 37, in threading._register_atexit(_python_exit) File "C:\Python\311_x64\Lib\threading.py", line 1527, in _register_atexit raise RuntimeError("can't register atexit after shutdown") RuntimeError: can't register atexit after shutdown

[rubenbaer]

Can you provide all package versions using pip freeze? This might not be a paho mqtt issue.

[KonstantinWaser]

absl-py==2.1.0 ace_tools==0.0 aiofiles==24.1.0 aiohttp==3.9.5 aiosignal==1.3.1 alembic==1.13.2 altgraph==0.17.4 annotated-types==0.6.0 anyio==4.3.0 appdirs==1.4.4 Appium-Python-Client==0.24 argon2-cffi==23.1.0 argon2-cffi-bindings==21.2.0 arrow==1.3.0 asttokens==2.4.1 astunparse==1.6.3 async-lru==2.0.4 attrs==23.2.0 azure-core==1.32.0 azure-identity==1.19.0 azure-storage-blob==12.24.1 Babel==2.14.0 bcrypt==4.1.2 beautifulsoup4==4.12.3 bidict==0.23.1 bleach==6.1.0 blinker==1.7.0 blis==0.7.11 bokeh==3.6.2 Brotli==1.1.0 bs4==0.0.2 cached-property==1.5.2 cachetools==5.3.3 catalogue==2.0.10 category-encoders==2.6.3 certifi==2024.2.2 cffi==1.16.0 charset-normalizer==3.3.2 chilkat==10.1.2 ci-info==0.3.0 click==8.1.7 cloudpathlib==0.18.1 cloudpickle==3.0.0 clr-loader==0.2.6 colorama==0.4.6 comm==0.2.2 confection==0.1.5 ConfigArgParse==1.7 configobj==5.0.9 configparser==7.1.0 construct==2.10.70 contourpy==1.2.1 cornac==1.18.0 cramjam==2.9.1 crate==0.35.2 cryptography==42.0.5 CTkMessagebox==2.5 CTkToolTip==0.8 customtkinter==5.2.2 cycler==0.12.1 cymem==2.0.8 darkdetect==0.8.0 debugpy==1.8.1 decorator==5.1.1 defusedxml==0.7.1 dill==0.3.8 dnspython==2.6.1 docutils==0.19 easyocr==1.7.1 EasyProcess==1.1 ebrec==0.0.1 entrypoint2==1.1 et-xmlfile==1.1.0 etelemetry==0.3.1 exchangelib==5.4.3 executing==2.0.1 fastai==2.7.15 fastapi==0.109.2 fastcore==1.5.45 fastdownload==0.0.7 fastjsonschema==2.19.1 fastparquet==2024.11.0 fastprogress==1.0.3 filelock==3.13.4 Flask==3.0.2 Flask-Cors==4.0.1 Flask-Login==0.6.3 Flask-Migrate==4.0.7 Flask-SQLAlchemy==3.1.1 Flask-WTF==1.2.1 flatbuffers==24.3.25 fonttools==4.51.0 fqdn==1.5.1 frozenlist==1.4.1 fsspec==2024.3.1 future==1.0.0 gast==0.5.4 gensim==4.3.3 geojson==3.1.0 gevent==24.2.1 geventhttpclient==2.2.1 google-auth==2.30.0 google-auth-oauthlib==1.2.0 google-pasta==0.2.0 greenlet==3.0.3 grpcio==1.62.2 h11==0.14.0 h2==4.1.0 h5py==3.11.0 hdbcli==2.19.21 hpack==4.0.0 html5lib==1.1 htmlmin==0.1.12 htmx==0.0.0 httpcore==1.0.5 httplib2==0.22.0 httptools==0.6.1 httpx==0.27.0 huggingface-hub==0.22.2 hyperframe==6.0.1 hyperopt==0.2.7 hypothesis==6.100.4 idna==3.6 ifaddr==0.2.0 ImageHash==4.3.1 imageio==2.34.2 imbalanced-learn==0.13.0 importlib-metadata==7.0.1 iniconfig==2.0.0 intel-openmp==2021.4.0 ipykernel==6.29.4 ipython==8.24.0 ipywidgets==8.1.5 IronPdf==2024.9.1.3 isodate==0.6.1 isoduration==20.11.0 itsdangerous==2.1.2 jaraco.classes==3.3.1 jedi==0.19.1 Jinja2==3.1.4 joblib==1.4.2 json5==0.9.25 jsonpointer==2.4 jsonschema==4.22.0 jsonschema-specifications==2023.12.1 jupyter-events==0.10.0 jupyter-lsp==2.2.5 jupyter_client==8.6.1 jupyter_core==5.7.2 jupyter_server==2.14.0 jupyter_server_terminals==0.5.3 jupyterlab==4.1.8 jupyterlab_pygments==0.3.0 jupyterlab_server==2.27.1 jupyterlab_widgets==3.0.13 kaitaistruct==0.10 keras==2.15.0 keyring==24.3.0 kiwisolver==1.4.5 langcodes==3.4.0 language_data==1.2.0 lazy_loader==0.4 libclang==18.1.1 lightfm==1.17 lightgbm==4.3.0 llvmlite==0.42.0 locust==2.26.0 looseversion==1.3.0 lxml==5.1.0 Mako==1.3.5 marisa-trie==1.2.0 Markdown==3.6 markdown-it-py==3.0.0 markdown2==2.4.13 MarkupSafe==2.1.5 matplotlib==3.8.4 matplotlib-inline==0.1.7 mdurl==0.1.2 mechanize==0.4.10 memory-profiler==0.61.0 missingno==0.5.2 mistune==3.0.2 mkl==2021.4.0 ml-dtypes==0.3.2 modbus-tk==1.1.3 more-itertools==10.2.0 mplcursors==0.6 mpmath==1.3.0 msal==1.31.1 msal-extensions==1.2.0 msgpack==1.0.8 mss==9.0.1 multidict==6.0.5 multimethod==1.10 murmurhash==1.0.10 mypy==1.9.0 mypy-extensions==1.0.0 mysql-connector-python==8.3.0 mystring==0.2.252 namex==0.0.8 nbclient==0.10.0 nbconvert==7.16.4 nbformat==5.10.4 nest-asyncio==1.6.0 networkx==3.2.1 nibabel==5.2.1 nicegui==1.4.28 ninja==1.11.1.1 nipype==1.8.6 nltk==3.8.1 node2vec==0.5.0 notebook==7.1.3 notebook_shim==0.2.4 numba==0.59.1 numpy==1.26.4 nvidia-ml-py==12.555.43 oauthlib==3.2.2 opencv-python-headless==4.10.0.84 openpyxl==3.1.2 opt-einsum==3.3.0 optree==0.11.0 oracledb==2.5.0 orjson==3.10.5 outcome==1.3.0.post0 overrides==7.7.0 packaging==23.2 pafy==0.5.5 paho-mqtt==2.1.0 pandas==2.2.1 pandas-profiling==3.2.0 pandas-stubs==2.2.0.240218 pandas_ui==0.1 pandasql==0.7.3 pandera==0.18.3 pandocfilters==1.5.1 paramiko==3.4.0 parso==0.8.4 pathlib==1.0.1 patsy==0.5.6 pdf2image==1.17.0 pdfminer.six==20231228 pdfplumber==0.11.2 pefile==2023.2.7 phik==0.12.4 pillow==10.4.0 platformdirs==4.2.1 plotly==5.22.0 pluggy==1.4.0 polars==0.20.21 portalocker==2.10.1 powerlaw==1.5 preshed==3.0.9 prometheus_client==0.20.0 prompt-toolkit==3.0.43 protobuf==4.25.3 prov==2.0.1 pscript==0.7.7 psutil==5.9.8 pure-eval==0.2.2 py-machineid==0.5.1 py4j==0.10.9.7 pyarmor==8.5.12 pyarmor.cli.core==6.5.3 pyarrow==19.0.0 pyasn1==0.5.1 pyasn1_modules==0.4.0 pyclipper==1.3.0.post5 pycountry==24.6.1 pycparser==2.21 pycryptodome==3.20.0 pycryptodomex==3.20.0 pydantic==1.10.17 pydantic_core==2.18.2 pydivert==2.1.0 pydot==3.0.2 pygame==2.6.1 Pygments==2.17.2 pyinstaller==6.4.0 pyinstaller-hooks-contrib==2024.1 PyJWT==2.8.0 pykeepass==4.0.6 pymongo==4.6.2 PyMuPDF==1.24.11 PyMuPDFb==1.24.6 PyNaCl==1.5.0 pyodbc==5.1.0 pyOpenSSL==24.0.0 pyparsing==3.1.1 PyPDF2==3.0.1 pypdfium2==4.30.0 pyperclip==1.8.2 pyplotlib==0.0.56 pyrfc==3.3.1 pyscreenshot==3.1 pyserial==3.5 PySocks==1.7.1 pyspark==3.5.1 pyspnego==0.10.2 pytesseract==0.3.10 pytest==8.0.2 python-bidi==0.4.2 python-dateutil==2.8.2 python-dotenv==1.0.1 python-engineio==4.9.1 python-json-logger==2.0.7 python-multipart==0.0.9 python-socketio==5.11.3 python-vlc==3.0.21203 pythonnet==3.0.4 pytz==2024.1 PyWavelets==1.8.0 pywin32==306 pywin32-ctypes==0.2.2 pywinpty==2.0.13 pyxnat==1.6.2 PyYAML==6.0.1 pyzmq==26.0.3 qgrid==1.3.1 rdflib==6.3.2 recommenders==1.2.0 referencing==0.35.1 regex==2024.4.16 requests==2.32.3 requests-mock==1.12.1 requests-ntlm==1.2.0 requests-oauthlib==2.0.0 retrying==1.3.4 rfc3339-validator==0.1.4 rfc3986-validator==0.1.1 rich==13.7.1 roundrobin==0.0.4 rpds-py==0.18.0 rsa==4.9 safetensors==0.4.3 schedule==1.2.1 scikit-image==0.24.0 scikit-learn==1.4.2 scikit-surprise==1.1.3 scipy==1.12.0 seaborn==0.13.2 selenium==3.5.0 selenium-wire==5.1.0 Send2Trash==1.8.3 shapely==2.0.4 shellingham==1.5.4 siemens-pac==0.20 simple-websocket==1.0.0 simplejson==3.19.3 six==1.16.0 sklearn-compat==0.1.3 smart-open==7.0.4 sniffio==1.3.1 sortedcontainers==2.4.0 soupsieve==2.5 spacy==3.7.5 spacy-legacy==3.0.12 spacy-loggers==1.0.5 SQLAlchemy==2.0.27 sqlalchemy-access==2.0.2 sqlalchemy-hana==1.3.0 srsly==2.4.8 sspilib==0.1.0 stack-data==0.6.3 starlette==0.36.3 statsmodels==0.14.2 sympy==1.12.1 tangled-up-in-unicode==0.2.0 tbb==2021.12.0 tenacity==8.3.0 tensorboard==2.15.2 tensorboard-data-server==0.7.2 tensorflow==2.15.1 tensorflow-estimator==2.15.0 tensorflow-intel==2.15.1 tensorflow-io-gcs-filesystem==0.31.0 termcolor==2.4.0 terminado==0.18.1 tf-slim==1.1.0 thinc==8.2.4 threadpoolctl==3.5.0 tifffile==2024.7.2 tinycss2==1.3.0 tokenizers==0.19.1 torch==2.3.1 torchvision==0.18.1 tornado==6.4 tqdm==4.66.2 traitlets==5.14.3 traits==6.3.2 transformers==4.40.0 trio==0.24.0 trio-websocket==0.11.1 typeguard==4.2.1 typer==0.12.3 types-docutils==0.20.0.20240227 types-Pillow==10.2.0.20240213 types-Pygments==2.17.0.20240106 types-python-dateutil==2.9.0.20240316 types-pytz==2024.1.0.20240203 types-setuptools==69.1.0.20240223 typing-inspect==0.9.0 typing_extensions==4.10.0 tzdata==2024.1 tzlocal==5.2 undetected-chromedriver==3.5.5 uri-template==1.3.0 urllib3==2.1.0 usersettings==1.1.5 uvicorn==0.30.1 vboxapi==1.0 vbuild==0.8.2 verlib2==0.2.0 visions==0.7.4 wasabi==1.1.3 watchfiles==0.22.0 wcwidth==0.2.13 weasel==0.4.1 webcolors==1.13 webencodings==0.5.1 websocket-client==1.8.0 websockets==12.0 Werkzeug==3.0.1 widgetsnbextension==4.0.13 winregistry==1.1.1 wrapt==1.14.1 wsproto==1.2.0 WTForms==3.1.2 xlrd==2.0.1 xyzservices==2025.1.0 yarl==1.9.4 youtube-dl==2021.12.17 zipp==3.17.0 zope.event==5.0 zope.interface==6.3 zstandard==0.22.0

[rubenbaer]

Thanks.

I just cooked up a more basic version. Note that this must run as a script and not in a REPL.

import threading

def run():
    import concurrent.futures.thread

threading.Thread(target=run).start()

There are no dependencies needed and thus it looks like a python issue. There might be other issues out there related to this.

To trigger this as written in the issue, requires some packages like dns.resolver, anyio and and others.

[PierreF]

The exception give you the hints of the problem: RuntimeError: can't register atexit after shutdown

You stopped the Python main thread (which I should be "after shutdown")

Slightly extended version of last sample

import time
import threading

def run():
    time.sleep(1) # simulate some expensive computation
    print("I'm called after main thread shutdown")
    import concurrent.futures.thread

threading.Thread(target=run).start()
print("main thread will shutdown now")

You should wait for the thread:

import threading

def run():
    import concurrent.futures.thread

t = threading.Thread(target=run)
t.start()
t.join()