cachier

cachier copied to clipboard

I have a use case where my cache is large, rarely written to, but read by many different threads at the same time thus causing timeout errors because only one thread can read the cache at the same time.

This PR allows multiple threads to read the cache using a shared lock but still preserve the thread-safety by requesting an exclusive lock on cache writes.

Therefore if multiple threads want to read the cache then they can do so simultaneously, but if only a single thread wants to write to it, then it needs to acquire an exclusive lock that prevents all other threads to read or write to the cache.

I tested locally for my use case, but would that potentially cause other issues that I didn't think of?

@shaypal5 I didn't run the tests yet, I'm waiting for early feedback :)

Hmm. That sounds useful, but assuming all tests pass, it should be optional.

Please figure out how to make this configurable. :) Do you want help with that? You can also take a look at current configurable options Cachier has.

@louismartin You can see tests above, right? :)

Yes I can see that they all failed x) I will try to fix this soon :)

Any news on this front, @louismartin ?

Hi @shaypal5 ,

Sadly I haven't been able to work on this and I don't know if I will have time to do it soon. Maybe we can close and I'll reopen if I find the time?

No need to close, on my side. It can wait as an open PR for now.

Hi @shaypal5 , I made the new feature configurable (although I enabled it by default). pickle_core tests pass on my side.

Ok. The repository has moved under a new GitHub organization, python-cachier, where you are a contributor.

Your PRs should now be able to successfully pull the secrets required by the test workflow to connect to the mongodb.com server used for tests, and thus you should be able to progress with PRs more independently (with working tests, pinging me only when you get to the code review and later merge phase, or if you need help, of course).

I've also added a flake8 lint job and fixed all flake8 errors except those in the pickle_core.py file, since you're working on it in you current 2 PRs.

PLEASE re-base both PRs over the latest head.

Please also fix all flake8 errors in this file in at least one of your PRs.

Thanks for contributing, friend! 😀

@louismartin What do you say? Contributing this should be easier now... :)

Closed after being stale for a long time. Please open a new PR if you with to contribute this feature at some point in the future.