pyshmht
pyshmht copied to clipboard
felix021
Huge memory usage
Hello I just ported it to Python 3.7 and made some tests to compare with Redis...
Performance is pretty good up to ~100,000 keys, however I am finding that the shared file seems unreasonably big, at ~500MB for that.
Is that expected?
I am also observing huge performance degradation at bigger sizes, probably because the file does not fit in RAM anymore...
Yes, the underlying hash table uses fixed key and value length in hashtable.c, therefore event if you only use a short key/value, it takes much space, maybe much more than expected. You can adjust bucket_size and max_key_size to reduce it, but be careful with your input.
This package is kind of experimental, and I didn't do much test on it, and it DO NOT use lock to avoid concurrency problems.
BE CAREFUL: this package is not for general purpose usage, it only accepts key < max_key_size and values < max_value_size. And although it's sharing memory based, it DO NOT use locks to avoid concurrency problem. It was designed for a former project, which had a write process, and many read process after the writer finished.
OK thanks anyway, it was an interesting experiment. Now I am working on a more general solution based on Boost::interprocess... If you don't mind, I will leave the issue opened so people do BE CAREFUL soon enough :-)