SQLite3 context inside the library

[jongarate]

I'm trying to open a ciphered database once built with YapDatabase using plain SQLCipher and I keep getting crashes whenever I try to set the decryption key using sqlite3_key(). These crashes happen because apparently the generated sqlite3 instance fails to have certain cache properties inside the binary tree. The following is the bit of code I'm using:

int flags = SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE | SQLITE_OPEN_NOMUTEX | SQLITE_OPEN_PRIVATECACHE;
int status = sqlite3_open_v2([databasePath UTF8String], &db, flags, NULL);
if (status == SQLITE_OK)
{
    NSData * dbPasswordData = [@"Th3S3cr3tPassw0rd" dataUsingEncoding:NSUTF8StringEncoding];
    status = sqlite3_key(db, [dbPasswordData bytes], (int)[dbPasswordData length]);
    if (status != SQLITE_OK) {
        NSLog(@"SQLite3 Error: %s ", sqlite3_errmsg(db));
        sqlite3_close(db);
    }
    status = sqlite3_exec(db, "SELECT count(*) FROM sqlite_master;", NULL, NULL, NULL);
    if (status != SQLITE_OK) {
        NSLog(@"SQLite3 Error: %s ", sqlite3_errmsg(db));
        sqlite3_close(db);
    }
}

Thing is, this crashes when used within an ordinary Controller/AppDelegate but works if copied within:

- (id)initWithPath:(NSString *)inPath
        serializer:(YapDatabaseSerializer)inSerializer
      deserializer:(YapDatabaseDeserializer)inDeserializer
           options:(YapDatabaseOptions *)inOptions
{

    /* Debugging chunk of code */
    int flags = SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE | SQLITE_OPEN_NOMUTEX | SQLITE_OPEN_PRIVATECACHE;
    int status = sqlite3_open_v2([databasePath UTF8String], &db, flags, NULL);
    if (status == SQLITE_OK)
    {
        NSData * dbPasswordData = [@"Th3S3cr3tPassw0rd" dataUsingEncoding:NSUTF8StringEncoding];
        status = sqlite3_key(db, [dbPasswordData bytes], (int)[dbPasswordData length]);
        if (status != SQLITE_OK) {
            NSLog(@"SQLite3 Error: %s ", sqlite3_errmsg(db));
            sqlite3_close(db);
        }
        status = sqlite3_exec(db, "SELECT count(*) FROM sqlite_master;", NULL, NULL, NULL);
        if (status != SQLITE_OK) {
            NSLog(@"SQLite3 Error: %s ", sqlite3_errmsg(db));
            sqlite3_close(db);
        }
    }
    /* End of chunk */


    return [self initWithPath:inPath
             objectSerializer:inSerializer
           objectDeserializer:inDeserializer
           metadataSerializer:inSerializer
         metadataDeserializer:inDeserializer
           objectPreSanitizer:NULL
          objectPostSanitizer:NULL
         metadataPreSanitizer:NULL
        metadataPostSanitizer:NULL
                      options:inOptions];
}

I'm aware pods aren't supposed to be edited but let's just keep that out of the discussion, this was done solely for debugging purposes. Point is, the SELECT query in the second excerpt of code works fine. And hence the question, What goes into the YapDatabase environment that makes such code work that can't outside of it?

==== EDITED ====

Just for additional context regarding the question, I should say that the call to:

YapDatabase * _database = [[YapDatabase alloc] initWithPath:databasePath serializer:NULL deserializer:NULL options:options];

Happens in the exact same Controller/AppDelegate place as in the direct sqlite3 calls in the first block of code posted here.

[chrisballinger]

Why are you opening it like that? I'm a little confused.

I think maybe this is what you're looking for: https://github.com/yapstudios/YapDatabase/blob/e05584ad54b827844866e840cb4aa0ffbe14a6f3/YapDatabase/YapDatabaseOptions.h#L157

Once you can open the database the usual way, use whatever solution you need to copy the contents to an unencrypted database.

[jongarate]

I think that perhaps the initial context I provided wasn't clear enough to understand the use case. Let's give it another go.

In an old version of my app, I used YapDatabase as the backend to store user data. In the upcoming release, I'm planning on getting rid of the YapDatabase pod dependency and storing data somewhere else. In this regard, and due to the burden called backwards compatibility, I need to perform a migration process for existing users for obvious reasons. To do so, instead of keeping the whole YapDatabase dependency forever and ever in the podfile, I thought that just bringing something like SQLCipher would get the job done to open the sqlite file, decrypt it and migrate data.

Now, coming back to the details of the topic, the shown sqlite3_* blocks behave differently when executed directly in whichever Controller/AppDelegate and when executed inside the YapDatabase.m class context. I'm wondering wether there's some contextual setting that makes sqlite3/sqlcipher behave differently inside the pod.

[chrisballinger]

Why not keep the dependency until enough users have migrated?

[jongarate]

Thanks Chris for keeping in touch.

In fact, the point here is to keep a dependency in place to give support to those less frequent users for catching up. But since the data contained in the database is so simple (just a handful of strings and numbers) and YapDatabase includes SQLCipher, why not simply keep SQLCipher ONLY and thus:

• Keep less pods in the Podfile for backwards compatibility purposes. (Ok, we would actually just change from 'YapDatabase/SQLCipher' to 'SQLCipher' but "less" here perhaps means "simpler")
• Ultimately lighten the resulting app binary by just keeping the necessary bits inside.

Actually, the migration process we're talking about has already been achieved and tested in a development context when the old version's database was not encrypted (and thus no need for SQLCipher). This issue arose as a result of the same process failing once encryption was enabled in the old version and we tried to open the database to move data to the new destination.

I understand that helping somebody moving away from one's project isn't commonplace but my guess is that the solution to this issue must be so simple that it's worth pinging for help.

[robbiehanson]

You may try copying some of the code from:

• configureDatabase
• configureEncryptionForDatabase

Disclaimer: I haven't tried it, but that would be my next guess.

[jongarate]

Thank you Robbie for joining the discussion.

That was exactly my approach as well. I debugged and reviewed the source code inYapDatabase to try to get an idea of what underlying steps where being performed each time such an environment was initialized (YapDatabase.m file, line 436 onwards if somebody want's to catch up) and hence came up with the sqlite3_* call sequence outlined in the initial comment:

1. sqlite3_open_v2()
2. sqlite3_key()
3. sqlite3_exec()

Live debugging here showed that no additional calls are needed to enable crypto besides the "key" call prior to executing arbitrary SQL calls with "exec".

However, the second call fails (some nasty binary tree free() issues arise) whenever I perform those in my regular controller code but work flawlessly if executed, for instance, inside the convenient call:

(id)initWithPath:(NSString *)inPath
        serializer:(YapDatabaseSerializer)inSerializer
      deserializer:(YapDatabaseDeserializer)inDeserializer
           options:(YapDatabaseOptions *)inOptions

but previous to it's return statement (check my first comment). This leads me to think that there's some environment setup going on with YapDatabase, either deliberately or due to some cumbersome setup of the Pod in my particular workspace. Could it be that the fact that I'm importing the library as a Pod rather than a copy-on-project triggers something else in Xcode?

[XabierGoros]

Hi guys,

I'm experiencing the same result as @jongarate when reusing a ciphered YapDatabase and opening it with SQLCipher. However, if I still use YapDatabase it is able to open the database file without any problem.

Is there any update on this issue?

Thanks

[robbiehanson]

The various PRAGMA settings may be required after opening the database (and prior to performing other actions). See the list of actions in [YapDatabase configureDatabase:]