ekzhang
How to delete 30 GB of game data from Firebase Realtime DB
Hello, this is a bit of a unique issue. Hello! We made Set with Friends back in 2020.
Anyway, it's 2024 and there are 10 million games played, and we've kept storing the game data in the database here forever. Unfortunately this is kind of bad for our bill, which is exceeding $160/month now and growing linearly.
This is a lot to pay for even with the community help and so I've been trying to figure out how to delete all the game data from old games to keep the site running despite all of this.
Unfortunately Firebase makes it very hard to delete data from your database, or to even iterate through it. So far I can only iterate through the data at all by taking a backup, unzipping it into a gigantic (32 GB) JSON file, and mmapping it before converting it into an ad hoc newline-delimited JSON Format.
# some file size stats from the backup
import mmap
f = open('large-dump.json', 'r+b')
mmapped_file = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ)
mmapped_file.find(b'"chats":{') 1
mmapped_file.find(b'"gameData":{') 471287388
mmapped_file.find(b'"games":{') 29039980018
mmapped_file.find(b'"lobbyChat":{') 31666955405
mmapped_file.find(b'"publicGames":{') 31679913677
mmapped_file.find(b'"stats":{') 31762024506
mmapped_file.find(b'"userGames":{') 31762024554
mmapped_file.find(b'"userStats":{') 32250896009
mmapped_file.find(b'"users":{') 32328120269
len(mmapped_file) 32451556703
The FIrebase Admin SDK does allow me to write scripts against the database, but unfortunately neither the web console ("import JSON"), CLI (firebase database:set /gameData <(echo '{}')), nor Admin SDK allows me to delete the /gameData key directly. It's too big. Well, I guess being too big was the problem in the first place, and that's why this is costing us a lot of money! 😅
Anyway this is turning into a bit of a headache, I would probably need to write a script to go through the backup, track which games have already been deleted in a separate database, then run a durable execution job to go through Firebase and delete all of the keys from the DB manually over the course of a week or so.
If anyone wants to help with this, feel free to message me, you would be doing a huge service in helping keep setwithfriends.com online and sustainable 🙏
firebase database:remove /gameData may work, see: https://firebase.blog/posts/2019/03/large-deletes-in-realtime-database/
All these old games contain old records and a history of this community. Please make a backup or another way to still view them available in some way.
All these old games contain old records and a history of this community. Please make a backup or another way to still view them available in some way.
I understand your frustration, I do not want to remove the games either, this is why I originally made the application keep old game states from the start. If you would like to keep the existing records you may offer to pay to run the site for $200/month. Let me know if this interests you.
I'm trying to reduce the impact of this, I could wipe the database entirely but that would remove all statistics from users and so on, so this is a compromise.
If you are interested in a backup, please email me at [email protected] — I'm happy to work with you to find a way to do this. Just please remember to be respectful and do not make demands! I have invested a lot of my own time and money (several thousand dollars) into this site and the community it supports.
Great! You seem to have missed the first email I wrote you (on Sep 4), did you get the one I sent you just now?