mod_wsgi
mod_wsgi copied to clipboard
GrahamDumpleton
Unable to locate python-home in mod_wsgi.4.9.2
Hello
I use to have a Django Rest API deployed with mod_wsgi, apache2 and python3.10. Everything worked fine. Without any updates or any modifications, mod_wsgi can't locate the virtual environment and the packages installed in it.
It seems that "python-home" is not recognised by wsgiDeamon.
What should I do to debug this error ?
Should I install the newest version of mod_wsgi4.9.4 ? If yes should I delete the current version ?
Thank you in advance for your help. Best regards
I would need to see your configuration but at a guess I would say that you have declared a daemon process group but not used WSGIProcessGroup directive or process-group option on WSGIScriptAlias to delegate that WSGI application to the daemon process group you set python-home. One other possibility is that the virtual environment was not created with the same X.Y version of Python as mod_wsgi was compiled for, they must match. See https://modwsgi.readthedocs.io/en/master/user-guides/virtual-environments.html for more details on setting up virtual environments.
Hi, Thank you for your response. Here is my configuration :
WSGIDaemonProcess myuser_wsgi user=myuser group=myuser python-home=/home/www/myuser/code/venvs/myvenv python-path=/home/www/myuser/code/server-django/ maximum-requests=10000
WSGIProcessGroup myuser_wsgi
WSGIScriptAlias / /home/www/myuser/code/server-django/main_app/wsgi.py
WSGIApplicationGroup %{GLOBAL}
WSGIPassAuthorization On
Alias /static /home/www/myuser/code/site/public
<Directory /home/www/myuser/code/server-django/main_app>
Require all granted
</Directory>
My virtual environment was created for python3.10 and I installed mod_wsgi4.9.2. Everything worked just fine.
Sorry for slow reply, was on holiday. Did you make progress on this?
Hello No worries. Unfortunately no I didn't manage to solve the issue. I'm reinstalling everything from scratch. If I stay with python 3.10 do I install mod_wsgi.4.9.2 or a higher version ? Thank you for you help.
Best regards
First verify what version mod_wsgi is actually compiled for and using.
- https://modwsgi.readthedocs.io/en/master/user-guides/checking-your-installation.html#python-installation-in-use
What do you get for the values referenced in the docs?
Can you also post the actual error messages you are getting and indicate where you are seeing them. Ensure that if posting anything from the Apache error logs that you capture the complete error message and some of the prior log messages from mod_wsgi as context. Right now am guess as you haven't said exactly what the errors were.
Also check permissions on your project code, the virtual environment, and any installed packages in the virtual environment to ensure they are going to be readable to user mod_wsgi runs your code as.
Hi Graham
I concur with this. Have a look pls:
(venv) root@:/var/www/html/jam/venv# pip install mod_wsgi --trusted-host pypi.org --trusted-host files.pythonhosted.org
Collecting mod_wsgi
Using cached mod_wsgi-4.9.4.tar.gz (497 kB)
Preparing metadata (setup.py) ... done
Building wheels for collected packages: mod_wsgi
Building wheel for mod_wsgi (setup.py) ... done
Created wheel for mod_wsgi: filename=mod_wsgi-4.9.4-cp310-cp310-linux_x86_64.whl size=726631 sha256=3890ad306ae7f9ad8cbf55963ed0b62f3f7323730e44aebd1f2f50d43d79cd1f
Stored in directory: /root/.cache/pip/wheels/51/a0/a2/38db21ea3bffe4d6c3b0e543ae6a6078b86a5aed98a7c013a5
Successfully built mod_wsgi
Installing collected packages: mod_wsgi
Successfully installed mod_wsgi-4.9.4
(venv) root@:/var/www/html/jam/venv# mod_wsgi-express module-config
LoadModule wsgi_module "/var/www/html/jam/venv/lib/python3.10/site-packages/mod_wsgi/server/mod_wsgi-py310.cpython-310-x86_64-linux-gnu.so"
WSGIPythonHome "/var/www/html/jam/venv"
(venv) root@:/var/www/html/jam# python -V
Python 3.10.6
(venv) root@:/var/www/html/jam# pip list
Package Version
---------- -------
jam.py 5.4.130
mod-wsgi 4.9.4
pip 22.0.2
setuptools 59.6.0
wheel 0.37.1
(venv) root@:/var/www/html/jam# tail -f /var/log/apache2/error.log
[Tue May 30 08:05:55.163986 2023] [http:trace4] [pid 2698:tid 140166886667840] http_request.c(438): [client 10.0.2.2:51027] Cookie: admin_session_8080=\\"/46W5InG0JWLC872hT/1/OTzazE=?info=eyJpcCI6ICIxMC4wLjIuMiIsICJ1dWlkIjogbnVsbCwgInVzZXJfaW5mbyI6IHsidXNlcl9pZCI6IG51bGwsICJyb2xlX2lkIjogbnVsbCwgInJvbGVfbmFtZSI6ICIiLCAidXNlcl9uYW1lIjogIiIsICJhZG1pbiI6IGZhbHNlfX0=\\"
[Tue May 30 08:05:55.164069 2023] [authz_core:debug] [pid 2698:tid 140166886667840] mod_authz_core.c(815): [client 10.0.2.2:51027] AH01626: authorization result of Require all granted: granted
[Tue May 30 08:05:55.164075 2023] [authz_core:debug] [pid 2698:tid 140166886667840] mod_authz_core.c(815): [client 10.0.2.2:51027] AH01626: authorization result of <RequireAny>: granted
[Tue May 30 08:05:55.164090 2023] [core:trace3] [pid 2698:tid 140166886667840] request.c(360): [client 10.0.2.2:51027] request authorized without authentication by access_checker_ex hook: /builder.html
[Tue May 30 08:05:55.164181 2023] [authz_core:debug] [pid 2698:tid 140166886667840] mod_authz_core.c(815): [client 10.0.2.2:51027] AH01626: authorization result of Require all granted: granted
[Tue May 30 08:05:55.164187 2023] [authz_core:debug] [pid 2698:tid 140166886667840] mod_authz_core.c(815): [client 10.0.2.2:51027] AH01626: authorization result of <RequireAny>: granted
[Tue May 30 08:05:55.164191 2023] [core:trace3] [pid 2698:tid 140166886667840] request.c(360): [client 10.0.2.2:51027] request authorized without authentication by access_checker_ex hook: /builder.html
[Tue May 30 08:05:55.198661 2023] [wsgi:info] [pid 2696:tid 140166886667840] mod_wsgi (pid=2696): Create interpreter '127.0.1.1:8081|'.
[Tue May 30 08:05:55.203630 2023] [wsgi:info] [pid 2696:tid 140166886667840] mod_wsgi (pid=2696): Adding '/var/www/html/jam/venv' to path.
[Tue May 30 08:05:55.206166 2023] [wsgi:info] [pid 2696:tid 140166886667840] [remote 10.0.2.2:51027] mod_wsgi (pid=2696, process='web', application='127.0.1.1:8081|'): Loading Python script file '/var/www/html/jam/wsgi.py'.
.
.
[Tue May 30 08:11:01.854595 2023] [http:trace4] [pid 2698:tid 140166878275136] http_filters.c(959): [client 10.0.2.2:51030] Connection: close
[Tue May 30 08:11:01.854620 2023] [http:trace4] [pid 2698:tid 140166878275136] http_filters.c(959): [client 10.0.2.2:51030] Content-Type: text/html; charset=iso-8859-1
[Tue May 30 08:11:02.259753 2023] [wsgi:info] [pid 2983:tid 140166916683648] mod_wsgi (pid=2983): Attach interpreter ''.
[Tue May 30 08:11:02.263727 2023] [wsgi:info] [pid 2983:tid 140166916683648] mod_wsgi (pid=2983): Adding '/var/www/html/jam/venv' to path.
[Tue May 30 08:11:02.268180 2023] [wsgi:info] [pid 2983:tid 140166916683648] mod_wsgi (pid=2983): Imported 'mod_wsgi'.
[Tue May 30 08:11:02.269696 2023] [wsgi:debug] [pid 2983:tid 140166886667840] src/server/mod_wsgi.c(9180): mod_wsgi (pid=2983): Started thread 0 in daemon process 'web'.
[Tue May 30 08:11:02.270120 2023] [wsgi:debug] [pid 2983:tid 140166861489728] src/server/mod_wsgi.c(9180): mod_wsgi (pid=2983): Started thread 3 in daemon process 'web'.
[Tue May 30 08:11:02.270179 2023] [wsgi:debug] [pid 2983:tid 140166869882432] src/server/mod_wsgi.c(9180): mod_wsgi (pid=2983): Started thread 2 in daemon process 'web'.
[Tue May 30 08:11:02.270195 2023] [wsgi:debug] [pid 2983:tid 140166878275136] src/server/mod_wsgi.c(9180): mod_wsgi (pid=2983): Started thread 1 in daemon process 'web'.
[Tue May 30 08:11:02.270226 2023] [wsgi:debug] [pid 2983:tid 140166853080640] src/server/mod_wsgi.c(9180): mod_wsgi (pid=2983): Started thread 4 in daemon process 'web'.
cat /etc/apache2/mods-enabled/wsgi.load
#LoadModule wsgi_module /usr/lib/apache2/modules/mod_wsgi.so
LoadModule wsgi_module "/var/www/html/jam/venv/lib/python3.10/site-packages/mod_wsgi/server/mod_wsgi-py310.cpython-310-x86_64-linux-gnu.so"
<VirtualHost *:80>
ServerAdmin webmaster@localhost
DocumentRoot /var/www/html/jam
ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined
LogLevel trace5
WSGIDaemonProcess web python-home=/var/www/html/jam/venv python-path=/var/www/html/jam/venv user=www-data group=www-data processes=1 threads=5
WSGIProcessGroup web
WSGIScriptAlias / /var/www/html/jam/wsgi.py
<Directory /var/www/html/jam>
Options +ExecCGI
SetHandler wsgi-script
AddHandler wsgi-script .py
Require all granted
<Files wsgi.py>
Require all granted
</Files>
</Directory>
<Directory /var/www/html/jam/static>
Require all granted
</Directory>
</VirtualHost>
Blank page.
OS: jammy jellyfish ubuntu
@platipusica You need to show how you configured mod_wsgi. Based on the log messages you are trying to use mod_wsgi daemon mode, yet you don't show that configuration so can't tell you what may be wrong with it.
@GrahamDumpleton this config is the same as before from about 2 years ago, did not change anything in the way the server was setup. BTW, I updated the comment with apache config. Is that enough?
Not sure how else to demonstrate the config. Pls advice how to to that. Cheers
Right now I can only assume that Debian/Ubuntu has broken their Python distribution such that it can no longer be used with embedded systems if you want to use a Python virtual environment. That or how Python virtual environments are now created and laid out is no longer compatible with what Python expects when used in an embedded environment. The Windows Python distribution has been broken like this for a very long time and a big hack is required when using Python C APIs to get it to work.
Can you explain how you created the virtual environment? Did you recreate the virtual environment recently, or was there a Python OS update, or change in Python version used? In other words, what was the change that triggered this?
Just this today, new OS and VM:
172 sudo apt install python3-virtualenv
173 virtualenv -p python3 venv
175 source venv/bin/activate
Everything works for the local development tho with VirtEnv. Can run gunicorn etc.
Hmm, looks like I wont be touching my site with 300+ Apps via WSGIScriptAliasMatch :)
When running gunicorn it isn't an embedded system, it is just a command line Python invocation, very different.
What you can try is to change to using:
python-path=/var/www/html/jam/venv/lib/python3.10/site-packages
What you had for python-path before wasn't actually needed and wouldn't have done anything as the virtual environment location is only meant to be in python-home.
This fiddle of telling python-path to look in site-packages may be enough to work around whatever they have broken.
Yep.
Did try that as well, no luck. Will recreate the virtenv again and keep you posted.
As always, appreciated your time.
Double check directory/file ownership/permissions on the virtual environment and ensure accessible to www-data user.
Also try some of the tests outlined in:
- https://modwsgi.readthedocs.io/en/master/user-guides/checking-your-installation.html#python-installation-in-use
to see if it yields anything interesting which may help.
Hi Graham,
looks like the simple wsgi is broken as well, with no mod_wsgi:
If I do this for example, a simple one line to 000-default.conf from: https://www.howtoforge.com/how-to-run-python-scripts-with-apache-and-mod-wsgi-on-ubuntu-22-04/
and access with: curl http://127.0.0.1/wsgi
nothing works.
What do you mean by "with no mod_wsgi"? The only way if not using mod_wsgi, to host Python code with Apache, would be as CGI scripts. So am a bit confused.
BTW, you since you use pip to install mod_wsgi, for testing while debugging this, could use:
mod_wsgi-express start-server .../path/to/wsgi.py
with the test hello world program. That way don't have to fiddle with main Apache configuration.
Yes,
mod_wsgi-express start-server is working with our App. No issues.
(venv) root@:/var/www/html/jam# mod_wsgi-express start-server --user=www-data --group=www-data wsgi.py Server URL : http://localhost:8000/ Server Root : /tmp/mod_wsgi-localhost:8000:0 Server Conf : /tmp/mod_wsgi-localhost:8000:0/httpd.conf Error Log File : /tmp/mod_wsgi-localhost:8000:0/error_log (warn) Operating Mode : daemon Request Capacity : 5 (1 process * 5 threads) Request Timeout : 60 (seconds) Startup Timeout : 15 (seconds) Queue Backlog : 100 (connections) Queue Timeout : 45 (seconds) Server Capacity : 20 (event/worker), 20 (prefork) Server Backlog : 500 (connections) Locale Setting : en_US.UTF-8
Reinstalled the server. No luck with Apache tho. Ie, with no virtualenv created:
[Wed May 31 04:23:34.698140 2023] [core:notice] [pid 4848:tid 140477774174080] AH00094: Command line: '/usr/sbin/apache2'
[Wed May 31 04:23:34.698969 2023] [wsgi:warn] [pid 5055:tid 140477774174080] (2)No such file or directory: mod_wsgi (pid=5055): Unable to stat Python home /var/www/html/jam/venv. Python interpreter may not be able to be initialized correctly. Verify the supplied path and access permissions for whole of the path.
Python path configuration:
PYTHONHOME = '/var/www/html/jam/venv'
PYTHONPATH = (not set)
program name = 'python3'
isolated = 0
environment = 1
user site = 1
import site = 1
sys._base_executable = '/usr/bin/python3'
sys.base_prefix = '/var/www/html/jam/venv'
sys.base_exec_prefix = '/var/www/html/jam/venv'
sys.platlibdir = 'lib'
sys.executable = '/usr/bin/python3'
sys.prefix = '/var/www/html/jam/venv'
sys.exec_prefix = '/var/www/html/jam/venv'
sys.path = [
'/var/www/html/jam/venv/lib/python310.zip',
'/var/www/html/jam/venv/lib/python3.10',
'/var/www/html/jam/venv/lib/python3.10/lib-dynload',
]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'
.
.
As seen PYTHONPATH is empty. This is before virtualenv tho, and after creating it, this test would work with mod_wsgi-express start-server AND Apache:
import sys
def application(environ, start_response):
status = '200 OK'
output = u'sys.path = %s' % repr(sys.path)
response_headers = [('Content-type', 'text/plain'),
('Content-Length', str(len(output)))]
start_response(status, response_headers)
return [output.encode('UTF-8')]
output: curl 127.0.0.1 sys.path = ['/usr/lib/python310.zip', '/usr/lib/python3.10', '/usr/lib/python3.10/lib-dynload', '/var/www/html/jam/venv/lib/python3.10/site-packages']
However, the App does not work with Apache.
When you run mod_wsgi-express it is still using Apache.
Now I don't remember you showing the error:
(2)No such file or directory: mod_wsgi (pid=5055): Unable to stat Python home /var/www/html/jam/venv. Python interpreter may not be able to be initialized correctly. Verify the supplied path and access permissions for whole of the path.
before.
This indicates that the directory doesn't exist, or the Apache user cannot access the directory. So check ownership and permissions again, including parent directories.
This system isn't using AppArmor is it? If it were configured then Apache may also be blocked from doing stuff when run as root, but would work when run mod_wsgi-express as normal user.
That error is after I reinstalled everything, so starting from a blank VM. So yes, expected this but a bit surprised with PYTHONPATH being empty.
AppArmor is not running. www-data user has no issue with permissions, can do "su - www-data" too, since enable it to test the access.
This is a worry:
Gateway Timeout
The gateway did not receive a timely response from the upstream server or application.
Apache/2.4.52 (Ubuntu) Server at 127.0.0.1 Port 8081
Really exhausted everything, and this is not even WSGIScriptAliasMatch, so not expecting this to work at all atm.
Since using daemon mode, to avoid any issues with Python being initialized in Apache child worker processes, ensure you are setting:
WSGIRestrictEmbedded On
at global scope outside of any VirtualHost directives. I imagine you may already have this, but still worth stating to be safe.
As to the error, the code is:
/*
* Is common to see people set the directory to an incorrect
* location, including to a location within an inaccessible
* user home directory, or to the 'python' executable itself.
* Try and validate that the location is accessible and is a
* directory.
*/
ap_log_error(APLOG_MARK, APLOG_INFO, 0, wsgi_server,
"mod_wsgi (pid=%d): Python home %s.", getpid(),
python_home);
#if !defined(WIN32)
rv = apr_stat(&finfo, python_home, APR_FINFO_NORM, p);
if (rv != APR_SUCCESS) {
ap_log_error(APLOG_MARK, APLOG_WARNING, rv, wsgi_server,
"mod_wsgi (pid=%d): Unable to stat Python home "
"%s. Python interpreter may not be able to be "
"initialized correctly. Verify the supplied path "
"and access permissions for whole of the path.",
getpid(), python_home);
}
else {
...
Do you see the message above that failure which shows Python home ....?
Not sure what trace5 does for LogLevel, can you set it to debug to verify see that message for Python home value. Otherwise not sure why not showing.
The gateway error is just because it is going to be the mod_wsgi daemon processes that aren't starting properly, so requests cannot be proxied across to them from Apache child worker processes which initially accept the request.
And just to confirm, creating shell using sudo as www-data and then running:
ls -las /var/www/html/jam/venv
works fine.
PYTHONPATH being empty is probably fine at that point as Python hasn't been initialized with configuration yet, so what it shows is more or less the default configuration.
Yes,
And just to confirm, creating shell using sudo as www-data and then running:
ls -las /var/www/html/jam/venv
works fine. I can source virtualenv, do pip install, etc. All with www-data user. No need to "sudo su - www-data" since using root user anyway, so all good with "su - www-data" for root.
The PYTHONPATH issue is now all good, since installed the virtualenv called venv.
Here is a vid from www-data user:
https://github.com/GrahamDumpleton/mod_wsgi/assets/9026100/e65ea690-f304-44d6-ac96-e98ff3227619
lynx would also work with the your test, but hangs indefinitely just like curl with the App.
Hi, IMO, gunicorn does not work either when used with virtualenv :
(venv) $gunicorn wsgi
[2023-05-31 07:54:55 +0000] [2615] [INFO] Starting gunicorn 20.1.0
[2023-05-31 07:54:55 +0000] [2615] [INFO] Listening at: http://127.0.0.1:8000/ (2615)
[2023-05-31 07:54:55 +0000] [2615] [INFO] Using worker: sync
[2023-05-31 07:54:55 +0000] [2616] [INFO] Booting worker with pid: 2616
[2023-05-31 07:54:55 +0000] [2616] [ERROR] Exception in worker process
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/gunicorn/arbiter.py", line 589, in spawn_worker
worker.init_process()
File "/usr/lib/python3/dist-packages/gunicorn/workers/base.py", line 134, in init_process
self.load_wsgi()
File "/usr/lib/python3/dist-packages/gunicorn/workers/base.py", line 146, in load_wsgi
self.wsgi = self.app.wsgi()
File "/usr/lib/python3/dist-packages/gunicorn/app/base.py", line 67, in wsgi
self.callable = self.load()
File "/usr/lib/python3/dist-packages/gunicorn/app/wsgiapp.py", line 58, in load
return self.load_wsgiapp()
File "/usr/lib/python3/dist-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
return util.import_app(self.app_uri)
File "/usr/lib/python3/dist-packages/gunicorn/util.py", line 384, in import_app
mod = importlib.import_module(module)
File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 883, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "/var/www/html/jam/wsgi.py", line 2, in <module>
from jam.wsgi import create_application
ModuleNotFoundError: No module named 'jam'
[2023-05-31 07:54:55 +0000] [2616] [INFO] Worker exiting (pid: 2616)
[2023-05-31 07:54:55 +0000] [2615] [INFO] Shutting down: Master
[2023-05-31 07:54:55 +0000] [2615] [INFO] Reason: Worker failed to boot.
(venv) $pip list |grep jam
jam.py 5.4.130
When used with NO virtualenv, then it works. So maybe Apache works too. Will try.
You would have to have pip installed gunicorn into the virtual environment. Doesn't look like you did and are using system package for it.
I don't usually recommend it, but try the activate_this method described in https://modwsgi.readthedocs.io/en/master/user-guides/virtual-environments.html for setting up a virtual environment.
Solved.
VirtualHost needs:
WSGIApplicationGroup %{GLOBAL}
Somehow this was not included in our old config.
The problem for OP might be python-path= in the config.
Thanks Graham for your time.
Really cannot see why adding WSGIApplicationGroup would make a difference. :-(
The WSGIScriptAliasMatch works as well. It does not work without WSGIApplicationGroup.