pokebase icon indicating copy to clipboard operation
pokebase copied to clipboard

Functions like get_data are inherently not thread safe

Open Z02X opened this issue 2 months ago • 0 comments

The current implementation of get_data directly calls save, which introduces a risk when used in multithreaded contexts. The disk I/O performed by save is not protected, but locking around it has serious performance implications.

In my own private implementation of a REST API client for PokeAPI in python (sorry just found out about this project), I solved this by separating in‑memory caching from disk persistence. My approach was:

  • Keep an in‑memory cache updated freely across threads.
  • Disable automatic disk saving during concurrent operations.
  • Manually invoke a disk save step once threads have finished, ensuring serialization and consistency.

One note is it would likely involve not using shelve as I believe that is not thread safe. But other features may be more suited.

With approval, I am willing to begin migrating my code to use pokebase, implement and test a option to disable auto‑save and give manual control.

Z02X avatar Dec 02 '25 20:12 Z02X