React-Native support

[Gerharddc]

I would like to use this library together with something like https://github.com/craftzdog/react-native-sqlite-2 (which provides a WebSQL compatible interface through SQLite) to get Indexeddb support on React-Native.

I was hopeing this would be easy and that something like the following should work:

import SQLite from 'react-native-sqlite-2';
import setGlobalVars from 'indexeddbshim';

let idb = {};
setGlobalVars(idb, { win: SQLite });
var { indexedDB, IDBKeyRange } = idb;

But this fails when trying to import indexeddbshim with the following error:

I think that maybe it is trying to do some fancy stuff because it might think it is running on Node. I passed the SQLite object as win in the config because it would not find openDatabase any other way.

What can I try to make this work?

[pwnbreak]

I managed to make it (almost) work with this:

import SQLite from 'react-native-sqlite-2';
import setGlobalVars from 'indexeddbshim/src/setGlobalVars';

import Dexie from 'dexie';

const win = {};

setGlobalVars(
  win,
  {
    checkOrigin: false,
    win: SQLite,
    deleteDatabaseFiles: false,
    useSQLiteIndexes: true,
  }
);

class MyDexie extends Dexie {
  constructor(name) {
    super(name, {
      indexedDB: win.indexedDB,
      IDBKeyRange: win.IDBKeyRange,
    });
  }
}

I say almost because Babel seems to be ignoring an arrow function in the output bundle:

Original:

Object.defineProperty(IDBVersionChangeEvent, Symbol.hasInstance, {
    value: obj => util.isObj(obj) && 'oldVersion' in obj && typeof obj.defaultPrevented === 'boolean'
});

Output:

Object.defineProperty(IDBVersionChangeEvent, typeof Symbol === 'function' ? Symbol.hasInstance : '@@hasInstance', {
    value: obj => util.isObj(obj) && 'oldVersion' in obj && typeof obj.defaultPrevented === 'boolean'
});

So there is a syntax error in Android, but at least it works in remote debugger for me.

[Gerharddc]

@nizkp Unfortunately testing with the remote debugger is not an option in this case because that makes all JS execute remotely in Chrome instead of through React-Native's own runtime. What that means is that normal Indexeddb should even work there without the shim.

Getting this Babel problem fixed could solve everything though.

[pwnbreak]

True, but it's actually working (data is stored in mobile via react-native-sqlite-2). But of course it's not useful if it can't run in the old Android JSC Engine. I think the problem must come from some custom Babel transform that react-native must be doing, or doing transforms in the wrong order. I'll try to investigate a little bit.

[brettz9]

Though the ES2015 preset should normally convert, FWIW, there is a Babel plugin for arrow functions: https://babeljs.io/docs/plugins/transform-es2015-arrow-functions/

[jeanregisser]

I'm using IndexedDBShim successfully with Dexie in a React Native project (iOS and Android) with the following code:

// file: configureDB.js

export default function configureDB() {
  // Make indexeddbshim happy with React Native's environment
  if (global.window.navigator.userAgent === undefined) {
    global.window.navigator = { ...global.window.navigator, userAgent: '' };
  }
  // Import after RN initilization otherwise we get an exception about process not being defined
  const SQLite = require('react-native-sqlite-2').default;
  global.window.openDatabase = SQLite.openDatabase;
  // babel-polyfill is Needed on Android otherwise indexeddbshim complains
  // about Object.setPrototypeOf missing
  require('babel-polyfill');
  require('indexeddbshim');
  global.shimIndexedDB.__setConfig({ checkOrigin: false });

  const Dexie = require('dexie');
  const db = new Dexie('my_db');

  // schema
  db.version(1).stores({
    friends: 'name,shoeSize'
  });

  return db;
}

// file: index.js
import React from 'react';
import {
  AppRegistry,
  View,
} from 'react-native';
import configureDB from './configureDB';

function setup() {
  db = configureDB();
  // [...]

  class App extends React.Component {
    render() {
      return (
          <View style={styles.container}>
             {...}
          </View>
      );
    }
  }
  return App;
}

AppRegistry.registerComponent('MyApp', setup);

This is a quite hackish and I wish we would have proper support for React Native directly in IndexedDBShim. Haven't found the time to make this happen myself though.

Also this setup is working (mostly) for my use case up until [email protected].

I say mostly as I have had incorrect results when querying multi-entry indexes with Dexie.

I thought those issues might be fixed by upgrading IndexedDBShim. But starting with [email protected] I get incorrect serialization/deserialization of objects containining undefined properties.

So I had to keep using v3.1.0.

Example:

// When saving the following object with IndexedDBShim: 
{ foo: "my value", bar: undefined }
// It is being stored within the sqlite DB as
{"foo":"my value","bar":null,"$types":{"bar":"userObject"}}
// And then when read back from the DB, I get this:
{ foo: "my value", bar: {} }

I tracked it down to typeson issues in this particular React Native env within the minified indexeddbshim bundle. Though when I manually run typeson stringifySync and parse with the same registered types in React Native I don't get any issue:

var Typeson = require('typeson');
var reg = require('typeson-registry');

// Simulates what happens in https://github.com/axemclion/IndexedDBShim/blob/6c4ed0694edefc37ce814140240d6f58f1e0f316/src/Sca.js

var structuredCloning = reg.presets.structuredCloningThrowing;

function traverseMapToRevertToLegacyTypeNames (obj) {
    if (Array.isArray(obj)) {
        return obj.forEach(traverseMapToRevertToLegacyTypeNames);
    }
    if (obj && typeof obj === 'object') { // Should be all
        Object.entries(obj).forEach(([prop, val]) => {
            if (prop in newTypeNamesToLegacy) {
                const legacyProp = newTypeNamesToLegacy[prop];
                delete obj[prop];
                obj[legacyProp] = val;
            }
        });
    }
}

var newTypeNamesToLegacy = {
    IntlCollator: 'Intl.Collator',
    IntlDateTimeFormat: 'Intl.DateTimeFormat',
    IntlNumberFormat: 'Intl.NumberFormat',
    userObject: 'userObjects',
    undef: 'undefined',
    negativeInfinity: 'NegativeInfinity',
    nonbuiltinIgnore: 'nonBuiltInIgnore',
    arraybuffer: 'ArrayBuffer',
    blob: 'Blob',
    dataview: 'DataView',
    date: 'Date',
    error: 'Error',
    file: 'File',
    filelist: 'FileList',
    imagebitmap: 'ImageBitmap',
    imagedata: 'ImageData',
    infinity: 'Infinity',
    map: 'Map',
    nan: 'NaN',
    regexp: 'RegExp',
    set: 'Set',
    int8array: 'Int8Array',
    uint8array: 'Uint8Array',
    uint8clampedarray: 'Uint8ClampedArray',
    int16array: 'Int16Array',
    uint16array: 'Uint16Array',
    int32array: 'Int32Array',
    uint32array: 'Uint32Array',
    float32array: 'Float32Array',
    float64array: 'Float64Array'
};

// console.log('StructuredCloning1', JSON.stringify(structuredCloning));
traverseMapToRevertToLegacyTypeNames(structuredCloning);
// console.log('StructuredCloning2', JSON.stringify(structuredCloning));

var TSON = new Typeson().register(structuredCloning);

var r = TSON.stringifySync({foo: "my value", bar: undefined });
console.log('==result', r);

var r2 = TSON.parse(r);
console.log('==result2', r2);

// it outputs the expected data:
==result {"foo":"my value","bar":null,"$types":{"bar":"undefined"}}
==result2 { foo: 'my value', bar: undefined }

So this effectively suggests the issue is within the minified IndexedDBShim bundle imported the way you see it above. Again proper support for React Native in IndexedDBShim would probably fix those side effects.

Hope it will help someone else.

[brettz9]

Great to have this info, thanks! I don't know when I may be able to get to looking at this to see what we can do to fix. Please keep us updated if you would if things change on the React Native side...

[brettz9]

@jeanregisser , can I ask if you also tested the non-minified bundle and whether that had problems for you too?

[brettz9]

I'd really like to better understand the issues in the latest IndexedDBShim regarding serialization/deserialization of objects with undefined as Typeson and typeson-registry tests are all passing (as are IndexedDBShim tests).

I suppose there is a chance though that the changes to Sca.js to revert legacy type names are not properly reverting pre-existing data with undefined?

[vladikoff]

This also works:

// IndexedDB hackery begins
Object.setPrototypeOf = Object.setPrototypeOf || function(obj, proto) {
  obj.__proto__ = proto;
  return obj;
}
// Make indexeddbshim happy with React Native's environment
if (global.window.navigator.userAgent === undefined) {
  global.window.navigator = { ...global.window.navigator, userAgent: '' };
}
const SQLite = require('react-native-sqlite-2').default;
global.window.openDatabase = SQLite.openDatabase;
require('indexeddbshim');
global.shimIndexedDB.__setConfig({ checkOrigin: false });

if you don't wanna do require('babel-polyfill');

shrugs

[brettz9]

In 3.6.0, the issue with setPrototypeOf should now be fixed. (Just don't set fullIDLSupport to true as tht will use Object.setPrototypeOf).

Note that as far as the previous comment, you don't need to polyfill userAgent if you invoke setGlobalVars. In that call, you can also add win: SQLite to your config (and you can forego a separate call to __setConfig by passing your config as the second argument to setGlobalVars).

But the latter example was a helpful reminder to check setPrototypeOf (which a dependency had been unconditionally adding, now fixed).

Anyways, feel free to report back on whether the fix works for everybody...

[vladikoff]

@brettz9 doing this:

const setGlobalVars = require('indexeddbshim');
setGlobalVars({
  win: SQLite
}, { checkOrigin: false });

Am I missing something?

[vladikoff]

Doing a log like this:

console.log('setGlobalVars', require('indexeddbshim'));

produces:

04-09 19:13:36.335  4379  4784 I ReactNativeJS: 'setGlobalVars', {}

so the setGlobalVars is not exported it seems

[brettz9]

@vladikoff : What do you get if you try this instead:

import setGlobalVars from 'indexeddbshim/src/setGlobalVars';
console.log(setGlobalVars);

[vladikoff]

@brettz9 the loading via /src/setGlobalVars doesn't work.

With import setGlobalVars from 'indexeddbshim'; I get this:

The way import statements are loaded, it seems that even having

if (global.window.navigator.userAgent === undefined) {
  global.window.navigator = { ...global.window.navigator, userAgent: '' };
}

above won't help to save it.

[brettz9]

How about

const setGlobalVars = require('indexeddbshim/dist/indexeddbshim-noninvasive.js');
console.log(setGlobalVars);

Btw, I can easily push a fix to check the existence of userAgent, just trying to get as much info first as possible...

[brettz9]

(If you wanted to go the import route, you'd need to find a way for importing module's like the Rollup plugin "rollup-plugin-node-builtins" (or equivalent for that environment) as our source is importing and relies on the Node module path, even for non-Node usage. That is the reason for the first error you reported in your last message.)

[zaptrem]

So what's the full currently established way of doing this?

[brettz9]

@zaptrem : See (or answer) my comments above to @vladikoff

[zaptrem]

I’m not sure which of those is experimentation/broken(due to some displaying crash screens)/intended for something else. My goal is to enable(?) the shim so that any other code that assumes IndexedDB exists can run with no modifications.

On Fri, Feb 7, 2020 at 9:44 PM Brett Zamir [email protected] wrote:

@zaptrem https://github.com/zaptrem : See (or answer) my comments above to @vladikoff https://github.com/vladikoff

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/axemclion/IndexedDBShim/issues/313?email_source=notifications&email_token=AAMJTRU6PTVBLMBMLAEUXB3RBYMBRA5CNFSM4ELGCFLKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELFHSHQ#issuecomment-583694622, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMJTRVE4TN7VUXXZ5AKJADRBYMBRANCNFSM4ELGCFLA .

[swittk]

Also interested in using an IndexedDB shim on React Native/Expo. (Just realized that Cloud Firestore wasn't doing Offline Persistence due to the lack of it)

I've tried doing what was mentioned in this thread https://github.com/firebase/firebase-js-sdk/issues/436 by @zwily (Gist here : https://gist.github.com/zwily/e9e97e0f9f523a72c24c7df01d889482), and sure enough, like others, it seems to work on iOS, and crashes on Android.

The crash shows a message I've identified to come from the function IDBObjectStore.prototype.__validateKeyAndValueAndCloneValue (inside '/indexeddbshim/dist/indexeddbshim-noninvasive.js')

and the error is thrown after the following checks

var _clonedValue = Sca.clone(value);

    key = Key.extractKeyValueDecodedFromValueUsingKeyPath(_clonedValue, me.keyPath); // May throw so "rethrow"

...

    if (key.failure) {

      if (!cursorUpdate) {

        if (!me.autoIncrement) {

          throw (0, _DOMException.createDOMException)('DataError', 'Could not evaluate a key from keyPath and there is no key generator'); 

        //--------> This here is the error that's happening on android

        }

I'm wondering if anyone could help shed some light on what is happening in this part of the code. The source is kind of full of semi-minified code and it's kind of hard to trace and understand them all. I also don't understand what seems to be causing the Android react native implementation to behave differently to the iOS version? Is there some class that is referenced somewhere that isn't fully implemented?

[zaptrem]

Also interested in using an IndexedDB shim on React Native/Expo. (Just realized that Cloud Firestore wasn't doing Offline Persistence due to the lack of it)

I've tried doing what was mentioned in this thread firebase/firebase-js-sdk#436 by @zwily (Gist here : https://gist.github.com/zwily/e9e97e0f9f523a72c24c7df01d889482), and sure enough, like others, it seems to work on iOS, and crashes on Android.

The crash shows a message I've identified to come from the function IDBObjectStore.prototype.__validateKeyAndValueAndCloneValue (inside '/indexeddbshim/dist/indexeddbshim-noninvasive.js')

and the error is thrown after the following checks

var _clonedValue = Sca.clone(value);

    key = Key.extractKeyValueDecodedFromValueUsingKeyPath(_clonedValue, me.keyPath); // May throw so "rethrow"

...

    if (key.failure) {

      if (!cursorUpdate) {

        if (!me.autoIncrement) {

          throw (0, _DOMException.createDOMException)('DataError', 'Could not evaluate a key from keyPath and there is no key generator'); 

        //--------> This here is the error that's happening on android

        }

I'm wondering if anyone could help shed some light on what is happening in this part of the code. The source is kind of full of semi-minified code and it's kind of hard to trace and understand them all. I also don't understand what seems to be causing the Android react native implementation to behave differently to the iOS version? Is there some class that is referenced somewhere that isn't fully implemented?

I think Android uses some special Facebook implementation of the JS Engine while iOS is required to use JavaScriptCore/WebKit.

[brettz9]

Thank you for reporting on what you have tried and where things are going wrong (the example code you have at https://gist.github.com/zwily/e9e97e0f9f523a72c24c7df01d889482 looks pretty sound, and may be helpful to others. (FWIW, I have added in a branch some code to avoid the need for the userAgent setting portion.) I'd like to help with this issue, but don't have time to dive into React/React-Native myself.

As far as the minified code, probably what you are seeing is from typeson/typeson-registry, the code responsible for encoding various object types into strings that can be stored in sqlite and then decoding them back to objects (to use the technical terminology, those that can be cloned by the "structured cloning algorithm"). I've started a branch with some code to move from grunt-browserify to grunt-rollup, as I think it is easier to work with, and should, if I can get it done, help me tweak the settings to avoid requiring the minified version in our non-minified builds (hopefully source maps would avoid the need, however).

The code you have excerpted (and much of the codebase) tries to follow the spec pretty closely, even in terminology as possible.

The code beginning Key.extractKeyValueDecodedFromValueUsingKeyPath is detailed at https://w3c.github.io/IndexedDB/#extract-a-key-from-a-value-using-a-key-path .

The subsequent checks are a part of our internal __validateKeyAndValueAndCloneValue which is called by IDBObjectStore add or put (and also by IDBCursor#update but that wouldn't get past the !cursorUpdate check, so looks like that is not it in your case). Presumably, you are therefore using an object store add or put, and Key.extractKeyValueDecodedFromValueUsingKeyPath returns a failure.

A failure can occur when the method evaluateKeyPathOnValueToDecodedValue is called.

If the key path is an array, the array items will be individually checked and if any fail, that call will fail. The other cases that can fail can be seen at https://github.com/axemclion/IndexedDBShim/blob/master/src/Key.js#L609-L613 (the lines preceding also set value).

These lines basically tell us that if the keypath sequence can't find a component in the sequence, then there will be a failure.

The issue could occur with _clonedValue if the typeson/typeson-registry cloning doesn't work in that environment for some reason. I'd therefore begin by introspecting to see what the value is. If it is a valid object that you expect at that point, then see what me.keyPath is, and whether it makes sense that that key path cannot be found on the given object. Without the object store being auto-increment, a DOMException is thrown as there is no way to determine what the key should be.

You can get usage help on Stackoverflow, but if there is a problem where cloning has not occurred properly, you can report that here (though it may need to go to typeson/typeson-registry, though I should also be able to help over there if that is the case).

[swittk]

Thanks for the helpful info :) I've been trying to identify the issue this past afternoon and was looking into evaluateKeyPathOnValueToDecodedValue's implementation, and even considering testing the native Array.isArray implementation on Android haha.

Scratch that : _clonedValue and value are the same.

What I've found is that when state updates are performed (I guess the add, or put function is called), the value parameter is simply an empty object! This is what is logged after var _clonedValue = Sca.clone(value); (Inside IDBObjectStore.prototype.__validateKeyAndValueAndCloneValue) Just before the crash. _clonedValue: {} value: {} keypath: batchId

Update : Empty object only occurs when 'add' is called; 'put' calls have objects with valid, expected keyPath property inside them.

[brettz9]

@swittk : I am guessing it is a bug in typeson or typeson-registry with the type of data you are trying to clone.

What version of indexeddbshim are you using? I would update to 6.0.1 (or current master is ok). There were a couple typeson-related upgrades which fixed one or two issues that also led to undefined. There's also one known issue left that has this problem: https://github.com/dfahlander/typeson/issues/11 .

However, it is puzzling to me that a simple object should have this issue. Is the value object {"clientId":"YI3pQPn90MSASrsyUvuR","updateTimeMs":1582120408087,"networkEnabled":true,"inForeground":false} just a plain object or are these on some special object type which also has those properties?

Since those encode/decode functions are just wrappers for typeson-registry with the structuredCloningThrowing preset, you will probably need to find some way to debug within or against typeson/typeson-registry. I don't know which workaround may work in your environment until I may get a non-minified version shipped, e.g., inlining the typeson/typeson-registry code, importing from its latest version, or using require statements (or script tags)--whatever React Native may support (or just include typeson-registry outside of indexeddbshim in React Native and see what happens when you clone your object there). Complicating this, I only recently fixed an issue and made a release for typeson-registry whereby it was not bundling the index.js source which is needed when requiring an ESM (as opposed to UMD) environment. While I have added reference to this new version in my indexeddbshim branch I'm working on, that is not yet released.

[swittk]

I'm so sorry, in my prior comment I wasn't logging the object correctly (logged clonedValue instead of _clonedValue).

I've since edited the post. I've found that value and _clonedValue are identical in all cases (So Sca.clone works, no problem).

However I've found that the crash occurs when the value provided is an empty object ( {} ), while keyPath is some value (keypath="batchId" to be exact; I guess it's something from Firestore)

At first start of the application; one 'add' operation is successful (This is logged in IDBObjectStore.prototype.add) key: undefined value:{"fbase_key":"__sak674766500","value":"674766500"}

_clonedValue, value, and keypath (fbase_key) are as expected inside __validateKeyAndValueAndCloneValue

after that there are multiple calls to 'put', which seem to behave normally as expected

Then just before the crash the condition referenced before happens (This is logged in IDBObjectStore.prototype.add) key: undefined value:{}

_clonedValue and value are {}, but keypath is 'batchId'

I'm not sure if I understand this correctly, but it seems either batchId came from somewhere unexpected within firebase, or the object itself isn't really a typical object.

However, I've compared the logs to the react-native iOS version (which do work) and found that the same log occurs in IDBObjectStore.add, however it doesn't crash...?

key:undefined, value:{} In __validateKeyAndValueAndCloneValue : _clonedValue: {}, value: {}, keypath: batchId

In the iOS version, after the curious 'add' with the empty object call, normal 'put' calls are executed after that with no issue, and no crash occurs.

update: Interestingly in iOS (where things don't crash); the add call with the empty object is immediately followed by a put call with a large object containing what looks like the data to be added. In this put call, me.keyPath is also "batchId", but the value object in the put call does have batchId inside of it.

This behavior probably coincides with creating a new document with no data in firestore then assigning data to it (to be specific in my case; creating a new DocumentReference, getting the id of it, then calling .set(value) to the DocumentReference). However, I still do not fully understand how batchId came to exist while the value object itself is completely empty..

[swittk]

I've since come to discover that the iOS and Android versions return identical results, and fail the same Key.extractKeyValueDecodedFromValueUsingKeyPath test (both return {"failure":true}).

The only difference is in the autoIncrement property; iOS has it being true, while Android has it being false, and thus it chugs along on one and fails on the other.

~~I'm wondering if this property (autoIncrement) is configurable with the current API. Trying to set it with setGlobalVars(window, {autoIncrement:true}) shows autoIncrement is not a valid configuration property.~~

Update: Seems autoIncrement is part of the firestore's invocation of createObjectStore, and looking in the firestore/firebase source's minified code; all instances of autoIncrement are set to !0 (true). So I'm not sure why the autoIncrement check is showing false here (on the me.autoIncrement check).

[swittk]

Just found another interesting thing in case it would help; I logged storeProperties.autoInc inside IDBObjectStore.__createInstance and the results were surprising;

on iOS almost all calls are autoInc:false, with very, very few calls having autoInc:true on Android it is the exact inverse; autoInc: true for almost all logs, with very few autoInc: false (and the last autoInc:false value is just right before the crash)

[swittk]

The logs show that the autoInc value provided to IDBObjectStore.__createInstance are the exact opposite in these two platforms .__. I don't know where to look for next; I'm guessing that the true/false values after the numerical values are due to the values being converted to Boolean somewhere, then copied around. But the initial numerical values being 0 instead of 1 and vice versa is confusing me.

[brettz9]

Seems autoIncrement is part of the firestore's invocation of createObjectStore, and looking in the firestore/firebase source's minified code; all instances of autoIncrement are set to !0 (true).

Looking at https://github.com/firebase/firebase-js-sdk/blob/542240d093ad54f3c8e5739dcb8449e8b9cab131/packages/firestore/src/local/indexeddb_schema.ts#L468-L470 , it appears that autoIncrement is missing from this case.

[brettz9]

Please note that as of version 6.1.0 just released, the code at https://gist.github.com/zwily/e9e97e0f9f523a72c24c7df01d889482#file-expo-firestore-persistence-hack-js-L31-L34 should no longer be needed.

I guess I'll keep this issue open in case someone can write up a section for a README explaining usage which is not tied to a particular library but may work for React Native (if such a thing is possible). But the README already details usage, so tbh, I'm not sure even this is necessary.

(The Firebase issue appears to be a bug in Firebase, so you might try submitting a PR which tries adding autoIncrement to that case where it is missing, and see if the devs might explore it further.)