Better approach of making a cache

[vdusek]

• This is a follow-up issue to the discussion in https://github.com/apify/crawlee-py/pull/82#discussion_r1548009445.
• Currently, we have our own implementation of LRU cache in crawlee/_utils/lru_cache.py.
• Let's do it in a more Pythonic way, maybe utilizing the built-in caching from functools std module (lru_cache decorator)?

[Azathoth-X]

Was reading on this and wanted to clear up somethings. The lru_cache from functools has no delete function for a given key, it just clears whole cache. Can you define what do you mean by pythonic? The cache implementation looks good as it.

Edit: I just saw the usage of this class and del wasn't used atleast in the python codebase. Please do tell me if it is also being used in the ts base. I will try to implement functools LRU . Will update you if there is any progress from my side.

[29deepanshutyagi]

i want to work on this issue ,kindly assign me @B4nan , if it's still opened

[janbuchar]

i want to work on this issue ,kindly assign me @B4nan , if it's still opened

We don't assign issues for hacktoberfest. If you want to work on this, open a PR. First mergeable one gets merged.

[vdusek]

Can you define what do you mean by pythonic?

Try using something from the standard libraries.

[Azathoth-X]

Hi @vdusek , Just wanted to ask do we need the delete individual type saved in cache function, in this class. I could only trace references of get, clear and creating key value pair for LRUcache class. Only place where the delete function is used is unit tests.

def test_del(lru_cache: LRUCache[int]) -> None:
    # Key error on non-existent key
    with pytest.raises(KeyError):
        del lru_cache['non-existent-key']
    # No error with existing key
    len_before_del = len(lru_cache)
    del lru_cache['a']
    assert len(lru_cache) == len_before_del - 1
    assert 'a' not in lru_cache

So. if using stl, is the delete function to be kept in the implementation or not?

[vdusek]

Hi @Azathoth-X, if the delete is used only in the tests, I believe we do not need it.