forking after importing bcolz generates segfaults in libpthread.so

[ssanderson]

On Ubuntu 14.04, Python 2.7.10 and Python 3.4, the following script causes segfaults in libpthread.so` in the child process during shutdown.

import os, bcolz
os.fork()

During child process shutdown, the following errors appear from dmesg:

Aug  7 16:15:08 Scott-Quantopian kernel: [223425.586867] python[15294]: segfault at 7f9fdd2969d0 ip 00007f9feffb75da sp 00007fffd5c84120 error 4 in libpthread-2.19.so[7f9feffae000+19000]

I can avoid these by manually calling _blosc_destroy in the parent before forking, but it's unclear whether this is reasonable and/or safe.

[ssanderson]

Note also that explicitly invoking _blosc_destroy in the child process always segfaults.

[esc]

I have not idea what is happening. Do you have any ideas why adding the _blosc_destory helps or why this scenario leads to a segfault?

[esc]

Also, why is it unclear whether this is reasonable and/or safe. What might make it unsafe?

[ssanderson]

@esc I think what's happening is that some of the global state set up by blosc_init gets carried over into the fork, but not all of it (in particular, I'm not sure how pthread mutexes interact with forking). When we call blosc_destroy in the child process, we try to tear down the only-partially-instantiated global state, resulting in unsafe memory accesses. Calling blosc_destroy in the parent before forking fixes this because we then have a clean slate in the child process.

Also, why is it unclear whether this is reasonable and/or safe. What might make it unsafe?

Presumably the calls to blosc_init and blosc_destroy happen for a reason, so I'm not sure whether, for example, I can continue to perform bcolz operations after calling destroy, or if I can call blosc_init again after calling destroy. From preliminary testing, bcolz continues to work happily after calling destroy, but I assume I'm playing with fire there.

[FrancescAlted]

Yes, forking is not a tested scenario, so I am not surprised to see this crashing. Probably the fix would be using the _ctx functions:

https://github.com/Blosc/c-blosc/blob/master/blosc/blosc.h#L184 https://github.com/Blosc/c-blosc/blob/master/blosc/blosc.h#L222

which do not require blosc_init/blosc_destroy.