perl-fuse
perl-fuse copied to clipboard
dpavlin
implement interpreter pool to reduce memory usage for threaded mode
FUSE treats threads as light-weight objects and creates/deletes them frequently. This leads to a new perl interpreter being created for all new threads, but never being reclaimed/reused when a thread goes away. So, a long running threaded mount continues to leak memory as new perl interpreters are continuously created.
To address that issue, this pull request implements an interpreter pool to satisfy requests when running in threaded mode. Threaded callbacks will attempt to use an inactive interpreter from the pool, or create a new one if none are available.
This pull request has been tested on the following versions of perl (all running on Fedora Core): 5.8.8 5.10.0 5.14.4 5.16.3
Because this is a threaded design change, I would ask that this pull request is tested on other architectures before it gets merged in to make sure I didn't overlook something.
That's my plan. I'll probably test it over the next few days; NetBSD and Mac OS X, I think, are the only non-Linux UN*X-like OSes I can think of that ship a threaded Perl interpreter. I'll probably have to dick around with FreeBSD for awhile to get a threaded build up without breaking my VM. Not sure if that's even possible with OpenBSD, since it's part of the base system. I'll see what I can figure out.
So far 'make test' seems to work fine on MacOS X 10.6 and NetBSD 6.0.2; is there anything in particular that should be done to beat on it?
I would try some simultaneous operations (such as multiple processes reading files at the same time) to exercise the interpreter pool and actually get it potentially swapping the active interpreters for threads. For my testing I usually spawn this command on multiple shells and then copy a file while it's going:
while true; do ls -alR lib; done;
I've now been running a couple mounts using this patch for 2 and a half weeks without having to remount them. They are currently using less memory than they were after a couple hours without this patch, and we haven't had any issues with using the mounts.
I'm satisfied that the code is stable on Fedora Core, not sure if you ever got around to trying any stress testing on other *nix flavors.
This one also fixes some deadlocks i had while using the threaded mode. THX! Any chance to get this upstream?
