ImportError: No module named billiard.forking with Buildout [now Windows only]

[reedobrien]

I have seen #26 and #10 but I am not using Django so the Django layout hack doesn't help.

I am attempting to use celery with a pyramid project that is managed with buildout. Starting celeryd leads to ImportError in the spawned billiard processes. A quick read through the code looks like it passes sys.path to the child processes. However, because the command that is run is "'from billiard.forking import main; main()'" it is trying to import billiard.forking before unpickling the path.

I can solve this by setting PYTHONPATH before calling:

prog = 'from billiard.forking import main; main()'
os.environ['PYTHONPATH'] = ':'.join(sys.path)
return [_python_exe, '-c', prog, '--billiard-fork']

If the forked process then sets sys.path to the unpickled parent one would it harm anything to just put it in billiard?

There may well be good reason not to. If so I would propose making it optional.

I was originally thinking of having a --with-buildout option, but there may be a more general reason for preserving sys.path in PYTHONPATH when calling.

I am not familiar enough with celery to know if how you pass options from celery to billiards. So as a guess, maybe something like this in celeryd's options:

...
Option('--preserve-syspath', action='store_true', default=False, dest='preserve_path'),
Option('--with-buildout', action='store_true', default=False, dest='preserve_path'),
...

and then checking for it properly here:

SHA: 3c75659a19e7cb472c62228486a26a6415ea4d6d

Thoughts?

[jjmurre]

+1 on this one, I have the same problem with buildout, celery and Django. Although, the suggested patch is not enough in the case of Django. In some weird way Django gets removed from the pythonpath later on, no idea how that happens.

[orientalperil]

+1 How do we use Celery with Buildout?

[ask]

I'm not sure about this, slamming all of sys.path into PYTHONPATH seems crude but I cannot see what other way it could solve this. I wonder about environment variable limits though, like how big can the value be?

[ask]

http://stackoverflow.com/questions/1078031/what-is-the-maximum-size-of-an-environment-variable-value

[ask]

The problem here is with buildout + multiprocessing. More specifically it's a problem on Windows, and with broker transports that uses the fork emulation (amqp and redis doesn't use this, so the problem is not present there) If you use a different pool it won't be a problem (e.g. eventlet, threads, gevent, solo)

rq doesn't use multiprocessing, so it doesn't have this problem. The introductory blog post is also full of shit :) Obviously Celery doesn't only target the 'big and serious' projects, this is why we support so many different brokers, it's also the reason for introducing the new API in 3.0.

The multiprocessing model has been the source of many problems for non-amqp/redis users. We could probably adapt the 'fork for every task' model for the problematic transports (database, etc). The performance implications, and the inability to remote control the processes is unlikely to be a problem for that user group anyway.

[orientalperil]

I'm not using Windows, but I get the same issue.

I'm in favour of the proposed change if you think that'll make it work.

[chchen]

I'm also seeing this issue. I'm currently working around the problem by setting the PYTHONPATH in my celeryd wrapper, but I'd rather sys.path get handed across.

[ask]

force_execv is now disabled by default since 3.0.15. It's not possible to do that on Windows though, since fork doesn't exist on that platform.

@chchen: Do you have this problem even if you use --no-execv?

[chchen]

Yes, it looks like I'm fine with 3.0.15! I saw the report and figured since it was still "open" that it hadn't be fixed yet.

Thanks for everything!

cc

On Thu, Mar 7, 2013 at 9:06 AM, Ask Solem Hoel [email protected]:

force_execv is now disabled by default since 3.0.15. It's not possible to do that on Windows though, since fork doesn't exist on that platform.

@chchen https://github.com/chchen: Do you have this problem even if you use --no-execv?

— Reply to this email directly or view it on GitHubhttps://github.com/celery/billiard/issues/28#issuecomment-14572839 .

"I want the kind of six pack you can't drink." -- Micah

[ask]

I'll update the subject to say Windows

[alexsilva]

I'm having the same problem with celery in Windows. The only option that works as expected is threadpool.