asgiref
asgiref copied to clipboard
ValueError: set_wakeup_fd only works in main thread
Hello everyone! Related to: 132:
Current situation
My environment is Windows Server 2016, Apache 2.4.39, Python 3.8.1 + mod_wsgi 4.7.1 + Django 3.0.3 + asgiref 3.2.3. I just updated from Django 2.2.10 to 3.0.3 and then my project started to crash with traceback:
[2020-02-20 17:58:04] | MainProcess | ERROR | 222 | django.request | log_response | Internal Server Error: /api/config/
Traceback (most recent call last):
File "C:\Program Files\Python38\Lib\site-packages\django\contrib\auth\middleware.py", line 57, in process_request
username = request.META[self.header]
KeyError: 'REMOTE_USER'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Program Files\Python38\Lib\site-packages\django\core\handlers\exception.py", line 34, in inner
response = get_response(request)
File "C:\Program Files\Python38\Lib\site-packages\django\utils\deprecation.py", line 93, in __call__
response = self.process_request(request)
File "C:\Program Files\Python38\Lib\site-packages\DiffApp\core\common\middleware.py", line 15, in process_request
super(CustomRemoteUserMiddleware, self).process_request(request)
File "C:\Program Files\Python38\Lib\site-packages\django\contrib\auth\middleware.py", line 62, in process_request
if self.force_logout_if_no_header and request.user.is_authenticated:
File "C:\Program Files\Python38\Lib\site-packages\django\utils\functional.py", line 224, in inner
self._setup()
File "C:\Program Files\Python38\Lib\site-packages\django\utils\functional.py", line 360, in _setup
self._wrapped = self._setupfunc()
File "C:\Program Files\Python38\Lib\site-packages\django\contrib\auth\middleware.py", line 24, in <lambda>
request.user = SimpleLazyObject(lambda: get_user(request))
File "C:\Program Files\Python38\Lib\site-packages\django\contrib\auth\middleware.py", line 12, in get_user
request._cached_user = auth.get_user(request)
File "C:\Program Files\Python38\Lib\site-packages\django\contrib\auth\__init__.py", line 180, in get_user
user = backend.get_user(user_id)
File "C:\Program Files\Python38\Lib\site-packages\django\contrib\auth\backends.py", line 161, in get_user
user = UserModel._default_manager.get(pk=user_id)
File "C:\Program Files\Python38\Lib\site-packages\django\db\models\manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "C:\Program Files\Python38\Lib\site-packages\django\db\models\query.py", line 411, in get
num = len(clone)
File "C:\Program Files\Python38\Lib\site-packages\django\db\models\query.py", line 258, in __len__
self._fetch_all()
File "C:\Program Files\Python38\Lib\site-packages\django\db\models\query.py", line 1261, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "C:\Program Files\Python38\Lib\site-packages\django\db\models\query.py", line 57, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "C:\Program Files\Python38\Lib\site-packages\django\db\models\sql\compiler.py", line 1142, in execute_sql
cursor = self.connection.cursor()
File "C:\Program Files\Python38\Lib\site-packages\django\utils\asyncio.py", line 19, in inner
event_loop = asyncio.get_event_loop()
File "C:\Program Files\Python38\lib\asyncio\events.py", line 636, in get_event_loop
self.set_event_loop(self.new_event_loop())
File "C:\Program Files\Python38\lib\asyncio\events.py", line 656, in new_event_loop
return self._loop_factory()
File "C:\Program Files\Python38\lib\asyncio\windows_events.py", line 310, in __init__
super().__init__(proactor)
File "C:\Program Files\Python38\lib\asyncio\proactor_events.py", line 632, in __init__
signal.set_wakeup_fd(self._csock.fileno())
ValueError: set_wakeup_fd only works in main thread
If I reboot Apache several time and if I lucky enough I can enter the proper thread (is is my suggestions) and app is running fine.
Now I have the temporal solution which is to add following lines to asgiref\__init__.py
(as it was suggested in 132:
if sys.platform == "win32" and sys.version_info >= (3, 8, 0):
asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
Do you know the cause of this error (specifically, that it is in 3.8.1?) As you said, #132 was fixed upstream in Python, and this looks like exactly that bug, so I'd want to make sure it's still in 3.8.1 and if so, maybe re-open the Python issue.
Unfortunately, I don’t. It started just after update to Django 3. But never happened in debug mode. Remote User Middlewares can be excluded from settings but nothing changes. I mean bug still appears and it is not problem of my custom middlewares
I'd appreciate it if you could absolutely verify you are on Python 3.8.1 by grabbing the value of sys.version
on the machine that's showing the bugs; that will get us the build number/date as well, so I can grab that exact version and try and replicate it on a Windows machine.
Sure!
Python 3.8.1 (tags/v3.8.1:1b293b6, Dec 18 2019, 23:11:46) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.version
'3.8.1 (tags/v3.8.1:1b293b6, Dec 18 2019, 23:11:46) [MSC v.1916 64 bit (AMD64)]'
OK, I'm very confused - your traceback's line numbers do, indeed, line up with a v3.8.1 release of Python, but the code at the end of your traceback is protected by this if
statement:
if threading.current_thread() is threading.main_thread():
(see it in context here: https://github.com/python/cpython/blob/v3.8.1/Lib/asyncio/proactor_events.py#L632)
Thus, it would be impossible to get to that statement unless you were already on the main thread. The other issue was triggering a different set_wakeup_fd
call that was unprotected (you can see the patch here: https://github.com/python/cpython/commit/cbf474c98e702d12c97cd16a1e44ede10ea52b5b)
Thus, the error you're having is not possible unless your threading
module is broken somehow. Do you have anything patching or altering it in some way?
I definetely have no patches on such crucial modules. To be completely sure that it is not my fault I am going to create an empty Django project and deploy it with Apache.
Also, a guy from stackoverflow reports about the same problem
Yeah, we just need to work out what is causing it. Apache may well be the cause - not sure - but either way I don't think this is a bug we can fix down in asgiref; hopefully we can find the cause and then get either the apache python integration or python itself to fix it.
Also, how are you deploying on apache - mod_wsgi?
Yes, I am using mod_wsgi 4.7.1 generated using python 3.8.1 Unfortunately, I delpoyed empty project and I can say that there is the same error even on empty Django site. I can get that error just reloading the page so it switchs from 500 to normal page and back
OK, my suspicion is that mod_wsgi
claims to a Python process that it is running in the main thread when, in fact, it is not (due to the main Apache thread being different to the main Python thread). I'll try to find some time later in the month to debug further, but for now I recommend you keep using the workaround you described above.
If you want to help me confirm, you could try switching away from Apache as the webserver and instead serving Django through something like gunicorn
.
OK, I just updated Python up to 3.8.2 and made comparison with waitress
1.1) Load Apache + mod_wsgi
1.2) Reload page -> 500 (set_wakeup_fd only works in main thread error)
1.3) After 1.2 next page refresh from time to time returns 500
2.1) Load waitress. Create server.py with
from waitress import serve
from mysite.wsgi import application
if __name__ == '__main__':
serve(application, port='80')
Then just:
python server.py
2.2) Reload page -> No errors 2.3) After first reload still no errors
Yeah, I suspect this is a bug in mod_wsgi somehow then. I'll have to look into it more when I get time.
I can produce the same bug with newer versions as following:
- Windows Server 2012 R2 Standard
- Apache 2.4.41
- Python 3.8.3
- PIP list:
asgiref==3.2.10
astroid==2.4.2
certifi==2020.6.20
cffi==1.14.0
colorama==0.4.3
cryptography==2.9.2
decorator==4.4.2
Django==3.0.7
django-templated-mail==1.1.1
djangorestframework==3.11.0
djoser==2.0.3
isort==4.3.21
lazy-object-proxy==1.4.3
mccabe==0.6.1
mod-wsgi==4.7.1
pycparser==2.20
pylint==2.5.3
python-telegram-bot==12.8
pytz==2020.1
six==1.15.0
sqlparse==0.3.1
toml==0.10.1
tornado==6.0.4
wfastcgi==3.0.0
wrapt==1.12.1
Error log:
[Tue Jul 07 11:42:51.788082 2020] [wsgi:error] [pid 6508:tid 1084] [client 78.180.215.26:53842] Exception ignored in: <function BaseEventLoop.__del__ at 0x000000BC070CE550>\r
[Tue Jul 07 11:42:51.788082 2020] [wsgi:error] [pid 6508:tid 1084] [client 78.180.215.26:53842] Traceback (most recent call last):\r
[Tue Jul 07 11:42:51.788082 2020] [wsgi:error] [pid 6508:tid 1084] [client 78.180.215.26:53842] File "c:\\python38\\lib\\asyncio\\base_events.py", line 656, in __del__\r
[Tue Jul 07 11:42:51.788082 2020] [wsgi:error] [pid 6508:tid 1084] [client 78.180.215.26:53842] self.close()\r
[Tue Jul 07 11:42:51.788082 2020] [wsgi:error] [pid 6508:tid 1084] [client 78.180.215.26:53842] File "c:\\python38\\lib\\asyncio\\proactor_events.py", line 679, in close\r
[Tue Jul 07 11:42:51.788082 2020] [wsgi:error] [pid 6508:tid 1084] [client 78.180.215.26:53842] signal.set_wakeup_fd(-1)\r
[Tue Jul 07 11:42:51.788082 2020] [wsgi:error] [pid 6508:tid 1084] [client 78.180.215.26:53842] ValueError: set_wakeup_fd only works in main thread\r
[Tue Jul 07 11:42:51.788082 2020] [wsgi:error] [pid 6508:tid 1084] [client 78.180.215.26:53842] Exception ignored in: <function BaseEventLoop.__del__ at 0x000000BC070CE550>\r
[Tue Jul 07 11:42:51.788082 2020] [wsgi:error] [pid 6508:tid 1084] [client 78.180.215.26:53842] Traceback (most recent call last):\r
[Tue Jul 07 11:42:51.788082 2020] [wsgi:error] [pid 6508:tid 1084] [client 78.180.215.26:53842] File "c:\\python38\\lib\\asyncio\\base_events.py", line 656, in __del__\r
[Tue Jul 07 11:42:51.788082 2020] [wsgi:error] [pid 6508:tid 1084] [client 78.180.215.26:53842] self.close()\r
[Tue Jul 07 11:42:51.788082 2020] [wsgi:error] [pid 6508:tid 1084] [client 78.180.215.26:53842] File "c:\\python38\\lib\\asyncio\\proactor_events.py", line 679, in close\r
[Tue Jul 07 11:42:51.788082 2020] [wsgi:error] [pid 6508:tid 1084] [client 78.180.215.26:53842] signal.set_wakeup_fd(-1)\r
[Tue Jul 07 11:42:51.788082 2020] [wsgi:error] [pid 6508:tid 1084] [client 78.180.215.26:53842] ValueError: set_wakeup_fd only works in main thread\r
[Tue Jul 07 11:42:51.788082 2020] [wsgi:error] [pid 6508:tid 1084] [client 78.180.215.26:53842] Exception ignored in: <function BaseEventLoop.__del__ at 0x000000BC070CE550>\r
[Tue Jul 07 11:42:51.788082 2020] [wsgi:error] [pid 6508:tid 1084] [client 78.180.215.26:53842] Traceback (most recent call last):\r
[Tue Jul 07 11:42:51.788082 2020] [wsgi:error] [pid 6508:tid 1084] [client 78.180.215.26:53842] File "c:\\python38\\lib\\asyncio\\base_events.py", line 656, in __del__\r
[Tue Jul 07 11:42:51.788082 2020] [wsgi:error] [pid 6508:tid 1084] [client 78.180.215.26:53842] self.close()\r
[Tue Jul 07 11:42:51.788082 2020] [wsgi:error] [pid 6508:tid 1084] [client 78.180.215.26:53842] File "c:\\python38\\lib\\asyncio\\proactor_events.py", line 679, in close\r
[Tue Jul 07 11:42:51.788082 2020] [wsgi:error] [pid 6508:tid 1084] [client 78.180.215.26:53842] signal.set_wakeup_fd(-1)\r
[Tue Jul 07 11:42:51.788082 2020] [wsgi:error] [pid 6508:tid 1084] [client 78.180.215.26:53842] ValueError: set_wakeup_fd only works in main thread\r
The only option to bypass this bug was downgrading Python from v3.8.3 to v3.7.8 because Error 500 kept to appear!!!
P.S. I'm really frustrated because this is my first project in Django and I shocked when I discovered this is a bug not a missing configuration by me!
BTW, I built mod_wsgi
using:
**********************************************************************
** Visual Studio 2019 Developer Command Prompt v16.6.3
** Copyright (c) 2020 Microsoft Corporation
**********************************************************************
[vcvarsall.bat] Environment initialized for: 'x64'
C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools>cl
Microsoft (R) C/C++ Optimizing Compiler Version 19.26.28806 for x64
Copyright (C) Microsoft Corporation. All rights reserved.
usage: cl [ option... ] filename... [ /link linkoption... ]
C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools>
I also ran into this error in my setup: Python 3.8.1, Apache v2.4.3, mod_authnz_sspi, mod_wsgi, Django 3.0.8
I was trying to incorporate Single Sign On that exists in my Organization's Windows network, into my Django web app. I was following the documentation on twigging Django settings on https://docs.djangoproject.com/en/3.0/howto/auth-remote-user/.
Following the instruction from the documentation, it tells me to add the following to settings.py:
MIDDLEWARE = [
'...',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.auth.middleware.RemoteUserMiddleware', # this causes "set_wakeup_fd only works in main thread" in django 3.0.8 but not in 2.2.14
'...',
]
AUTHENTICATION_BACKENDS = [
'django.contrib.auth.backends.RemoteUserBackend',
]
With these settings, the "set_wakeup_fd only works in main thread" error started happening when I start Apache, and try to access the django website.
I was able narrow down the setting that produced this error, which is 'django.contrib.auth.middleware.RemoteUserMiddleware'. When I comment out this setting, accessing the django website has no error.
It's good to note that when I downgraded django 3.0.8 to django 2.2.14, the "set_wakeup_fd only works in main thread" error never happened with all the above settings.
all newest version (apache, python, django on Windows 10)
I am running into this issue since a few weeks - funny: it started to pop up irregularly and now it is all the time.
When does it appear?
- It appears always the first time a query is made on a model or a Model.objects.create is called. It seem that calling a model is the first acces to signal.set_ .....
- It only appears on Apache for Windows. Apache on Ubuntu is running without any issues
- It is this line in "asyncio/proactor_events.py" with the call to signals. ...
if threading.current_thread() is threading.main_thread(): signal.set_wakeup_fd(self._csock.fileno())
that causes the problem. I can comment it out and the app will run normally. But I have no idea what is the consequence of commenting it out. 4) it seems to get stuck after a interrupted call to a Django Site. As if the Django code did not run to the end and leaves some "sand" in the threading stuff. If I then call another virtual host site on the same apache server with just a index.html "hello world" then I can call again my Django app and it starts properly. 5) in fact: it always constantly appears after a server restart, and it gets away by first calling the "hello world" site.
sorry if it sounds not very clear but it is not very clear to me.
It's September 2020 I have Python 3.8.3 x64 bit / Apache 2.4.46 / Windows Server 2016 / mod_wsgi (installed to Python not Apache) and still have the problem. I did like @Razenstein - commented the part and that's it for now. Working finally. For my development Windows 10 I haven't found such problems.
I would suggest in mod_wsgi config, if on Windows, of using something like:
WSGIScriptAlias / /some/path/wsgi.py application-group=%{GLOBAL}
The use of application-group
on WSGIScriptAlias
will force pre-loading of the WSGI script file. This preloading should occur in the main thread.
What I suspect may be occurring is that the threading
module isn't being imported during normal Python interpreter initialisation, and is only subsequently being imported the first time a request arrives, which will be on an external thread (non main thread), as a result the main_thread()
gets initialised to be the external thread and something blows up.
Anyway, this is guess as I don't grok the overall issue as brain not working well enough to get my head around it.
If this solves the problem, I would suggest this is a bug in CPython in that the threading
module assumes that it is always first imported by the main thread and never another thread.
# Create the main thread object,
# and make it available for the interpreter
# (Py_Main) as threading._shutdown.
_main_thread = _MainThread()
...
def main_thread():
"""Return the main thread object.
In normal conditions, the main thread is the thread from which the
Python interpreter was started.
"""
return _main_thread
This is fine if the threading
module is always imported during interpreter/sub interpreter initialisation, but would be a problem if it isn't guaranteed to be.
I would suggest in mod_wsgi config, if on Windows, of using something like:
WSGIScriptAlias / /some/path/wsgi.py application-group=%{GLOBAL}
The use of
application-group
onWSGIScriptAlias
will force pre-loading of the WSGI script file. This preloading should occur in the main thread.
It seems to me that your suggestion solves the issue. I just added WSGIScriptAlias and WSGIApplicationGroup lines to my config:
..
WSGIScriptAlias / my_path_to_wsgi/wsgi.py application-group=%{GLOBAL}
<VirtualHost app.com:443>
WSGIApplicationGroup %{GLOBAL}
...
Include ${APP_HOME}/conf/common.conf
</VirtualHost>
I report no errors with these changes. But if I remove it I get errors right after Apache reload. Need to test for a while.
You don't strictly need WSGIApplicationGroup
as the application-group
argument on WSGIScriptAlias
overrides that.
WSGIScriptAlias / my_path_to_wsgi/wsgi.py application-group=%{GLOBAL}
inside the VirtualHost definition is working fine on Windows 10 like suggested above! Thanks for the Solution!!
FWIW, there will be a more permanent workaround for this CPython behaviour in mod_wsgi version 4.8.0 when released.
I would suggest in mod_wsgi config, if on Windows, of using something like:
WSGIScriptAlias / /some/path/wsgi.py application-group=%{GLOBAL}
The use of
application-group
onWSGIScriptAlias
will force pre-loading of the WSGI script file. This preloading should occur in the main thread.What I suspect may be occurring is that the
threading
module isn't being imported during normal Python interpreter initialisation, and is only subsequently being imported the first time a request arrives, which will be on an external thread (non main thread), as a result themain_thread()
gets initialised to be the external thread and something blows up.Anyway, this is guess as I don't grok the overall issue as brain not working well enough to get my head around it.
If this solves the problem, I would suggest this is a bug in CPython in that the
threading
module assumes that it is always first imported by the main thread and never another thread.# Create the main thread object, # and make it available for the interpreter # (Py_Main) as threading._shutdown. _main_thread = _MainThread() ... def main_thread(): """Return the main thread object. In normal conditions, the main thread is the thread from which the Python interpreter was started. """ return _main_thread
This is fine if the
threading
module is always imported during interpreter/sub interpreter initialisation, but would be a problem if it isn't guaranteed to be.
This approach worked for me. Thanks for the answer.
I would suggest in mod_wsgi config, if on Windows, of using something like:
WSGIScriptAlias / /some/path/wsgi.py application-group=%{GLOBAL}
The use of
application-group
onWSGIScriptAlias
will force pre-loading of the WSGI script file. This preloading should occur in the main thread.What I suspect may be occurring is that the
threading
module isn't being imported during normal Python interpreter initialisation, and is only subsequently being imported the first time a request arrives, which will be on an external thread (non main thread), as a result themain_thread()
gets initialised to be the external thread and something blows up.Anyway, this is guess as I don't grok the overall issue as brain not working well enough to get my head around it.
If this solves the problem, I would suggest this is a bug in CPython in that the
threading
module assumes that it is always first imported by the main thread and never another thread.# Create the main thread object, # and make it available for the interpreter # (Py_Main) as threading._shutdown. _main_thread = _MainThread() ... def main_thread(): """Return the main thread object. In normal conditions, the main thread is the thread from which the Python interpreter was started. """ return _main_thread
This is fine if the
threading
module is always imported during interpreter/sub interpreter initialisation, but would be a problem if it isn't guaranteed to be.
This simply worked! Thanks a lot! And thanks for the explanation! I am using XAMPP to run the Apache server. I noticed that when I try to access my Django app, # of ports running Apache arbitrarily increases. In the browser it says "ValueError at /" since I ran with DEBUG=True. However, the page loads after refreshing the browser. So I kind of had the idea that the problem would be in the wsgi configuration. All in all, your explanation really tells what might have happened.
Note that mod_wsgi 4.8 with workaround for issue in CPython was released. Ensure using latest version of mod_wsgi. If still have issues it may be something different that you are seeing.
Hi,
I have the same error with python 3.9.5. I am making a windows service. I use cherrypy as webserver hosting django. If I run the service in debug mode, no problem. But if I run it in normal mode, I get the following error:
Traceback (most recent call last):
File "C:\My-Auslieferung-v2\resource\server\bpServer3\bpService3.py", line 25, in SvcDoRun
self.server.run()
File "C:\My-Auslieferung-v2\resource\server\bpServer3\BpServer3Launcher.py", line 80, in run
from bpServer3.wsgi import application
File "C:\My-Auslieferung-v2\resource\server\bpServer3\bpServer3\wsgi.py", line 17, in <module>
application = get_wsgi_application()
File "C:\My-Auslieferung-v2\python-3.9.5-embed-amd64\Lib\site-packages\django\core\wsgi.py", line 12, in get_wsgi_application
django.setup(set_prefix=False)
File "C:\My-Auslieferung-v2\python-3.9.5-embed-amd64\Lib\site-packages\django\__init__.py", line 24, in setup
apps.populate(settings.INSTALLED_APPS)
File "C:\My-Auslieferung-v2\python-3.9.5-embed-amd64\Lib\site-packages\django\apps\registry.py", line 122, in populate
app_config.ready()
File "C:\My-Auslieferung-v2\resource\server\bpServer3\jobs\apps.py", line 15, in ready
from jobs.views import x
File "C:\My-Auslieferung-v2\resource\server\bpServer3\jobs\views.py", line 53, in <module>
cleanUpTask.doOnStartCleanUp()
File "C:\My-Auslieferung-v2\resource\server\bpServer3\jobs\bpserver\CleanUpTask.py", line 25, in doOnStartCleanUp
for job in runningJobs:
File "C:\My-Auslieferung-v2\python-3.9.5-embed-amd64\Lib\site-packages\django\db\models\query.py", line 280, in __iter__
self._fetch_all()
File "C:\My-Auslieferung-v2\python-3.9.5-embed-amd64\Lib\site-packages\django\db\models\query.py", line 1324, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "C:\My-Auslieferung-v2\python-3.9.5-embed-amd64\Lib\site-packages\django\db\models\query.py", line 51, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "C:\My-Auslieferung-v2\python-3.9.5-embed-amd64\Lib\site-packages\django\db\models\sql\compiler.py", line 1173, in execute_sql
cursor = self.connection.cursor()
File "C:\My-Auslieferung-v2\python-3.9.5-embed-amd64\Lib\site-packages\django\utils\asyncio.py", line 19, in inner
event_loop = asyncio.get_event_loop()
File "asyncio\events.py", line 639, in get_event_loop
File "asyncio\events.py", line 659, in new_event_loop
File "asyncio\windows_events.py", line 310, in __init__
File "asyncio\proactor_events.py", line 632, in __init__
ValueError: set_wakeup_fd only works in main thread of the main interpreter
Modifying asgiref like proposed in the first post fixed my problem.