Kombu filesystem transport not thread safe

[harlowja]

From looking it over it appears that this transport isn't thread safe.

I've hit errors like http://paste.openstack.org/show/102226/ a few times and am thinking these class of errors are due to files getting moved by more than one thread. Perhaps some basic locking is in order?

[harlowja]

I've been able to see this where the file length that is read (by a consumer) ends up being zero length; quite odd, even if I use the processed_folder and store_processed options. Any idea? It doesn't occur all the time (but I'm guessing this is due to threads being non-determinstic...)

[harlowja]

A few more examples with a test program that seems to cause this issue...

http://paste.ubuntu.com/8184078/

https://github.com/harlowja/BrokenTest is the test program (at test.py)

Pretty reliably get that error...

[harlowja]

Also the virtualenv that I am using:

Babel==1.3 amqp==1.4.6 anyjson==0.3.3 importlib==1.0.3 iso8601==0.1.10 kombu==3.0.21 ordereddict==1.1 pytz==2014.4 six==1.7.3 wsgiref==0.1.2

[pran1990]

I ran the BrokenTest Josh posted above too, and saw similar errors. http://paste.ubuntu.com/8190131/

[ask]

It probably isn't thread safe, and that would be low on my priorities to fix

[harlowja]

Understandable that its low priority, but seems at least justified to document that?

[mpenalver]

Attached the version we use, which also avoids the copy of the msgs to tmp. Credit to https://github.com/benediktschmitt/py-filelock. filesystem.py.tar.gz

[jhofeditz]

Thanks! @mpenalver I also have this problem and will try your fix. Can you submit it as a PR?