crystal-sqlite3 icon indicating copy to clipboard operation
crystal-sqlite3 copied to clipboard

Add support for serialize/deserialize

Open SamantazFox opened this issue 2 years ago • 2 comments

Could you please add support for sqlite3_serialize and sqlite3_deserialize?

As a downstream app, that would allow us to manipulate database in memory only, removing the need to write data on disk!

SamantazFox avatar Feb 15 '24 21:02 SamantazFox

Can you give me an example of how would you use this? First time I see those functions.

In particular sqlite3_deserialize seems it will not play nice with the semantics of the db pool:

The sqlite3_deserialize(D,S,P,N,M,F) interface causes the database connection D to disconnect from database S and then reopen S as an in-memory database based on the serialization

Either the operations are just available as C functions (which they can happen outside crystal-sqlite3 if needed) or we need some reasonable Crystal api for these operations. An example would be a great starting point. I see there are many options and I am not sure what would be best for the general use case.

bcardiff avatar Feb 16 '24 19:02 bcardiff

Our use case would be importing/exporting user data from/to a sister app.

As an example, here is our current code to import said user data. As you can see, we have to create a file on disk, open it, read the user data, close, and make sure it is deleted. https://github.com/iv-org/invidious/blob/1e6ec605e88d1874e1b8b99294312a3c51f07beb/src/invidious/user/imports.cr#L292-L335

The goal would be to replace that code with something like this:

DB.open "sqlite3::memory" do |db|
  db.deserialize(file_io) # or maybe this?: sqlite3_deserialize(db, file_io)

  watched_raw = db.query_all("SELECT url FROM streams", as: String)
  subscriptions_raw = db.query_all("SELECT url FROM subscriptions", as: String)
  
  # The in-memory database is safely closed here
end

# Use the imported data

SamantazFox avatar Feb 16 '24 21:02 SamantazFox