iiilx
Configureable Pool Size
in main reading max_conn from settings would be nice so the module can be configured from django settings.py. The default of 3 is alright compared to just psycogreen, but with 15-30 threads (little difference between these two) my micro VPS performance doubled to 90 dynamic requests/s.
Curiously gunicorn and postgres stay busy after the test at 90+ req/s. This coincides with the level of requests at which performance chaotically grinds to a halt. gunicorn_defunct shows up a lot in top under these conditions. Willing to help diagnose this further, but I need some ideas on what to look for.
I haven't tested this much at all with web requests. I did test this with a management command that is used to run a separate service that makes use of connection pooling. I'm not sure why gunicorn and postgres would stay busy after the test is done. Perhaps looking at progres logs would help diagnose any issues.
I wasn't aware that running as a separate command was a desired way to implement. Can you elaborate on the designed use case so far?
On Sun, Dec 2, 2012 at 9:36 PM, Ben Lee [email protected] wrote:
I haven't tested this much at all with web requests. I did test this with a management command that is used to run a separate service that makes use of connection pooling. I'm not sure why gunicorn and postgres would stay busy after the test is done. Perhaps looking at progres logs would help diagnose any issues.
— Reply to this email directly or view it on GitHubhttps://github.com/iiilx/django-psycopg2-pool/issues/4#issuecomment-10940616.
Ideally this would work for either situation. I just haven't tested it really for web requests. What I'm doing is running a websocket server in a management command, mainly to have access to Django models. In this case, however, connections have to be put back manually. With web requests, the db backend however should put it back into the pool.
Regarding the load testing you've done, are the queries complex at all? unoptimized queries can cause a lot of switching between greenlets and chaos. I've definitely seen this with eventlet.
The queries are only designed to be a light test-case. Sessions and Django CMS all running. Seems to work okay.
I think I might be getting a little closer to the problem. Whenever httperf times out, it appears gunicorn and postgresql are still doing their thing unaware that the client is no longer there. This is probably an Nginx/gunicorn config error, not related to the DB backend.