jazzband
Multiprocessor support
An elegant idea from Scott Lowder:
I watched your presentation at DjangoCon a few weeks ago, you mentioned you were having some trouble with DB issues with multiprocess testing. I am very familiar with that pain, I have an 8+ hour test suite with crazy ad-hoc DB modification all over the tests. I never got the built-in multiprocess testing working against all our code, although I tried some crazy hacks to get it going, such as using a custom DB connection class that checks to see whether your current test database is specific for the current process, and getting a fresh copy if not. I did however get it working another way: 8+ hours is now 22 minutes, spread over 3 hosts and 40 cores, without having to figure out how each test was uniquely incompatible. The solution was to go multiprocess before importing Django, so that I can uniquely patch TEST_NAME in the database settings in the initializer method of each process. Same technique for any other shared resources, like KEY_PREFIX for Memcache settings.
This is what we did in disqus/mule but you then have to deal with the rest of the network resources and it gets painful fast.
We actually could probably do it now as we simply spin up a custom redis instance per run (which also works for things like memcache), but quickly became complicated.
On Oct 12, 2012, at 1:46 PM, Erik Rose [email protected] wrote:
An elegant idea from Scott Lowder:
I watched your presentation at DjangoCon a few weeks ago, you mentioned you were having some trouble with DB issues with multiprocess testing. I am very familiar with that pain, I have an 8+ hour test suite with crazy ad-hoc DB modification all over the tests. I never got the built-in multiprocess testing working against all our code, although I tried some crazy hacks to get it going, such as using a custom DB connection class that checks to see whether your current test database is specific for the current process, and getting a fresh copy if not. I did however get it working another way: 8+ hours is now 22 minutes, spread over 3 hosts and 40 cores, without having to figure out how each test was uniquely incompatible. The solution was to go multiprocess before importing Django, so that I can uniquely patch TEST_NAME in the database settings in the initializer method of each process. Same technique for any other shared resources, like KEY_PREFIX for Memcache setti ngs.
— Reply to this email directly or view it on GitHub.
Does anyone have a documented way of getting multiprocess to work with django-nose? The runner seems to accept my --processes argument, but it's ineffectual; i.e., MultiProcessTestRunner.run never seems to actually execute.
I am manually specifying a custom runner with TEST_RUNNER = 'foo_runner'. Could that be throwing it off?
I'm in the same situation as @jamesob - has there been any progress on this in the last few months?
I went to mule to see how they handled it, but I haven't been able to figure out yet (e.g., https://github.com/disqus/mule/issues/1)
Has anyone here made progress on this? I'd even be interested in a proof of concept (which is what I am using mule for until I can figure out the aforementioned issue.)
That was actually just a Disqus specific thing that we used for connection routing. You could basically replace a lot of that with how Django internally bootstraps connection sin a test suite.
On Monday, January 27, 2014 at 10:30 PM, onceuponapriori wrote:
I went to mule to see how they handled it, but I haven't been able to figure out yet (e.g., disqus/mule#1 (https://github.com/disqus/mule/issues/1)) Has anyone here made progress on this? I'd even be interested in a proof of concept (which is what I am using mule for until I can figure out the aforementioned issue.)
— Reply to this email directly or view it on GitHub (https://github.com/django-nose/django-nose/issues/100#issuecomment-33454386).
