[Discussion] Support for creating an in-memory YapDatabase instance

[joelekstrom]

We use YapDatabase to drive a large part of our UI. Sometimes, we need to show views with quite temporary data (all the data needs to be refetched on the next launch anyway) - but most our view controllers/datasources are already wired to present data using YapDatabase with various view mappings and extensions.

Right now we insert the data into YapDatabase and empty the affected collections on the next insert, but it feels like it would make much more sense if it was possible to open an in-memory only YapDatabase, and pass that connection to temporary view controllers. I was pondering since you can run SQLite in memory, perhaps it wouldn't be too hard to adapt YapDatabase to support that.

Another reason for why this would be great is that it would be much easier to change/refactor/rename the classes of these temporary objects, since we wouldn't have to worry that we renamed a class that might be accidentally deserialized on the next app launch. No need to worry about migrations if we know that these objects are never actually stored on disk anyway.

I'd be happy to provide a PR for this, but I'd like to discuss it with the maintainers first - is it a bad idea? Are there constraints that would make it much harder than I think it would be? Is it against the ideas of the project? etc. Thanks!

[robbiehanson]

Is it against the ideas of the project?

Nope. It sounds like a great idea. If memory serves me correctly, the only reason YapDatabase doesn't support this is because SQLite didn't support it. That is, when I was first designing YapDatabase (back in 2012) SQLite didn't seem to support the combination of:

• in-memory only database
• WAL (write-ahead logging)

I remember that I didn't fully understand why the 2 didn't work together. And ultimately attributing the incompatibility (in my mind) to the fact that WAL support was fairly new at the time.

I did some quick Internet searches, and didn't find anything discussing this problem. So perhaps it's been fixed since then?

[joelekstrom]

Thanks for the input! I'll investigate any potential issues related to WAL and in-memory databases. I'll also take a closer look to see if I can find any potential problems regarding this with how YapDatabase is implemented.

In the meantime, I would appreciate some input on how the API could look if this was implemented. Some suggestions:

1. Add a property to YapDatabaseOptions, to specify that it is in-memory only con: will make the path argument ambiguous
2. Add init methods that don't take a path for in-memory databases con: will lead to loads of init-methods
3. Treat a nil path as in-memory con: will lead to unexpected behaviour if you fail to create a path and accidentally send nil
4. Check if path is :memory:, which is the SQLite convention of creating in-memory databases. con: the name path might be misleading if it supports other things than an actual path.

I prefer number 4 - but I'm happy to discuss or see other ideas!

[joelekstrom]

After some investigation, ~I believe a temporary database is the cleanest way to achieve what I'm looking for here~. EDIT: Since it doesn't seem like different connections can share a temporary database, it's not fit for the YapDatabase architecture. An in-memory database with a shared cache is probably the best bet. Some initial testing says that YapDatabase does not support it right now.

[robbiehanson]

Some initial investigation & thoughts:

Since every :memory: connection is different/unique, YapDatabase won't be able to share the connection. So that's out.

However, the docs for sqlite3_open_v2 mention that we can use URI filenames. From what I gather via the in-memory docs, this means we'd need to pass a path parameter like file:some_unique_id_here?mode=memory&cache=shared.

"the same in-memory database can be opened by two or more database connections"

However, from the URI docs:

"In order to maintain full backwards compatibility for legacy applications, the URI filename capability is disabled by default."

"URI filenames can be enabled for individual database connections by including the SQLITE_OPEN_URI bit in the set of bits passed as the F parameter to sqlite3_open_v2(N,P,F,V)."

That last bit of information is news to me. I've never tried that before.

[robbiehanson]

I would appreciate some input on how the API could look if this was implemented

Assuming we still need a unique name (for the URI-based filename):

• Add init methods that takes a name for in-memory databases.
• Add some kind of this-is-an-in-memory-database flag somewhere, and use to disable the checkpoint stuff in YapDatabase.m (i assume?)

[mickeyl]

Any updates here?

[joelekstrom]

Yo! I’ve spent a few hours on this but haven’t managed to get it working. I don’t really understand why though, the docs seem pretty clear about the special file names.

Anyway, since creating a temp database file kind of solves the problem, I couldn’t justify spending too much time on it, but might pick it up again if more people need it!

[mickeyl]

Ok, I guess I'm stumbling at the same issue as you. I have created the following simple patch:

diff --git a/YapDatabase/YapDatabase.m b/YapDatabase/YapDatabase.m
index 5957fe8..e95fcbd 100644
--- a/YapDatabase/YapDatabase.m
+++ b/YapDatabase/YapDatabase.m
@@ -695,6 +695,10 @@ static int connectionBusyHandler(void *ptr, int count) {
        // as we will be serializing access to the connection externally.

        int flags = SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE | SQLITE_OPEN_NOMUTEX | SQLITE_OPEN_PRIVATECACHE;
+    if ( [databasePath hasPrefix:@"file:"] )
+    {
+        flags |= SQLITE_OPEN_URI;
+    }

        int status = sqlite3_open_v2([databasePath UTF8String], &db, flags, NULL);
        if (status != SQLITE_OK)
diff --git a/YapDatabase/YapDatabaseConnection.m b/YapDatabase/YapDatabaseConnection.m
index 4bf1f95..13a7554 100644
--- a/YapDatabase/YapDatabaseConnection.m
+++ b/YapDatabase/YapDatabaseConnection.m
@@ -274,6 +274,10 @@ static int connectionBusyHandler(void *ptr, int count)
                        // as we will be serializing access to the connection externally.

                        int flags = SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE | SQLITE_OPEN_NOMUTEX | SQLITE_OPEN_PRIVATECACHE;
+            if ( [database.databasePath hasPrefix:@"file:"] )
+            {
+                flags |= SQLITE_OPEN_URI;
+            }

                        int status = sqlite3_open_v2([database.databasePath UTF8String], &db, flags,
                                                     [database->yap_vfs_shim_name UTF8String]);

and opened the database with the path file:database?mode=memory&cache=shared.

Although both the two calls to sqlite3_open_v2 as well as the sqlite3_exec in createTables() seem to go through, I get errors along the way because of yap2 not being found.

Could passing database->yap_vfs_shim_name possibly interfere? @robbiehanson Do you have an idea what could go wrong?