tinydb
tinydb copied to clipboard
msiemens
TinyDB and a webserver
Hi there! Really love the TinyDB project since it feels like MongoDB in sqlite3 form :) Finally an easy and lightweight document db for python.
I'm using TinyDB in combination with the Streamlit framework to save some cryptocurrency data but for some reason after a couple of inserts and shutting down/starting the Streamlit app(which has a Tornado webserver running under the hood) my db.json always gets corrupted and it spews a bunch of JSONDecodeErrors like so:
When I then view the db.json file it shows me a bunch of NULLs like the insert was incorrect or the DB connection didn't shut down properly when closing the rest of the app:
Not entirely sure what I can do to fix it, my db.inserts are 2 strings every time and I create the db connection as a singleton.
When I delete the db.json it works again for a while. I just tried a couple of inserts again with a new db.json and ran into the JSONDecodeErrors again after starting and stopping the application about 20 times, VS Code thinks the db.json is unreadable now:
Maybe TinyDB isn't supposed to be used in combination with a webserver that essentially runs in the same python script?
Update: this might be my mistake actually not putting the Streamlit import at the top. Will test thoroughly and see if it happens again.
@frankhuurman even it you could fix it by moving the import, this should not happen. Is anything else accessing the db file at the same time? If the data is corrupted, then the json lib "should" error before even writing. The line where the error occurs is where the data is loaded from the file and then parsed. So it get's corrupted somewhere between writing and reading.
*I think
(Also checkout BetterJSONStorage as an alternative Storage Type ;))
Maybe TinyDB isn't supposed to be used in combination with a webserver that essentially runs in the same python script?
I guess it depends on how the webserver is implemented 🙂 TinyDB has basically no protection against data races when multiple threads try to write data simulatneously. Even though you created the database object as a singleton, it seems like the webserver runs in multiple threads which in some circumstances run database code in parallel. At least that matches the behavior you described.
A simple solution would be to add a lock around the part of your code that calls TinyDB so only one thread tries to write data at the same time.
Does that solve your problem?
