Add equivalents for goToFirst(), goToLast(), goToPrev(), goToNext() and goToRange() which can return also value.

[degifted]

Since there is no cursor function in LMDB API to retrieve only key, node-lmdb wrapper functions goToPrev(), goToNext() and goToRange() have to get both key and value, then return only key and omit the value. So, if you need to build an iterator where you need to get both keys and values, you need to call cursor.getCurrent_Type_() which adds some overhead. I propose to add the following functions to node-lmdb API:

[key, value] = goToFirstString() [key, value] = goToFirstBinary() [key, value] = goToFirstNumber() [key, value] = goToFirstBoolean()

[key, value] = goToLastString() [key, value] = goToLastBinary() [key, value] = goToLastNumber() [key, value] = goToLastBoolean()

[key, value] = goToPrevString() [key, value] = goToPrevBinary() [key, value] = goToPrevNumber() [key, value] = goToPrevBoolean()

[key, value] = goToNextString() [key, value] = goToNextBinary() [key, value] = goToNextNumber() [key, value] = goToNextBoolean()

[key, value] = goToRangeString(range) [key, value] = goToRangeBinary(range) [key, value] = goToRangeNumber(range) [key, value] = goToRangeBoolean(range)

This family of functions would also return null at the end of the cycle, and array of key and value otherwise. I understand that in most cases key and value are in the same memory page but this still would have some performance benefit if values are >4kb size.

[Venemo]

@degifted Thank you for the suggestion! What do you think, what exactly would be the performance benefit of this change?

[degifted]

Well in my case performance benefit was up to 2..4% depending on database size, so for me it was worth hacking. It turned out that dealing with v8 objects and especially allocating could be quite expensive. So I came to slightly different solution. I pass an array as a first argument of a goTo* function (i.e., where the callback should be) and return data as 0 element of that array. No any allocation is needed inside the C++ code. I am attaching my diff in case you are interested.

diff.txt

[degifted]

Example of use:

var tmpArray = []; for (var index = cursor.goToRangeBinary(key, tmpArray); index != null; index = cursor.goToPrevBinary(tmpArray)) { value = tmpArray[0]; }

[Venemo]

@degifted What is the array for?

[degifted]

@Venemo Since we cannot pass a reference to a string or number as a function argument, I create an array outside of the loop and then get an acquired value as '0' element of that array. I believe this is a fasted way of getting both key and value from a single function call.

[degifted]

This is probably relevant just to my case since I have to deal with hundreds of millions rows in the database

[Venemo]

@degifted Ah, so you basically use the array instead of creating an object. I think I get it now. Maybe I can also suggest a better way of doing it.

[Venemo]

@degifted Can you give me a performance test which shows the performance advantages of the array approach?

[degifted]

@Venemo Actually the test is quite simple:

var cnt = 1;
for (let id = cursor.goToFirst(); id != null; id = cursor.goToNext()) {
  if (!(cnt++%1000000)){console.timeEnd('million');console.time('million');console.log(cnt-1);}

  var data = cursor.getCurrentBinary();
}

I tested it on a database with ~150 millions of rows of 1..10Kb size. Here are results:

1. getCurrentBinary() 1000000 million: 1231.739ms 2000000 million: 1197.662ms 3000000 million: 1222.589ms 4000000 million: 1228.700ms 5000000 million: 1238.249ms 6000000 million: 1206.375ms 7000000 million: 1223.173ms 8000000 million: 1214.191ms 9000000

2. getCurrentBinaryUnsafe() 1000000 million: 957.309ms 2000000 million: 922.274ms 3000000 million: 933.243ms 4000000 million: 924.707ms 5000000 million: 933.442ms 6000000 million: 951.363ms 7000000 million: 947.090ms 8000000 million: 935.834ms 9000000

3. Array approach 1000000 million: 831.230ms 2000000 million: 827.088ms 3000000 million: 832.944ms 4000000 million: 825.808ms 5000000 million: 832.019ms 6000000 million: 827.016ms 7000000 million: 861.788ms 8000000 million: 830.044ms 9000000

4. No Data fetch 1000000 million: 750.358ms 2000000 million: 750.114ms 3000000 million: 753.595ms 4000000 million: 766.521ms 5000000 million: 747.345ms 6000000 million: 744.624ms 7000000 million: 745.001ms 8000000 million: 758.390ms 9000000

[degifted]

During the work on my own database layer I came to an intermediate 'quick and dirty' implementation of the missing functions. Here are the list of them. Hope it is self explaining.

Txn wrapper:
getUInt32
putUInt32

Cursor wrapper:
goToKeyUInt32
putUInt32
goToFirstUInt32Dup
goToLastUInt32Dup
goToNextUInt32Dup
goToPrevUInt32Dup
goToFirstUInt32
goToLastUInt32
goToNextUInt32
goToPrevUInt32
goToRangeUInt32
goToNextNoDup
goToPrevNoDup

These methods allowed to implement indexed search over duplicated keys of integer and string types. I also added UInt32 key type since existing Number type allocates 8 bytes which is unnecessary for an index table as it only addresses rows in the main data table. All goTo*UInt32() functions return value, whereas standard goTo*() return key, as usual.

[degifted]

After some thought I think that implementing such a lengthy cycles in Javascript is a bad idea anyway. I probably have to move it to the C++ layer

[Venemo]

@degifted I'll look into your patches and merge those of them that I think would be useful to others too.

I also added UInt32 key type since existing Number type allocates 8 bytes

There was already an uint32 key type :)

[degifted]

@Venemo Sorry I meant data type, which is also a key type in the main table in my case. Not a big deal but it saves 4 bytes.

Local<Value> valToNumber(MDB_val &data) {
    return Nan::New<Number>(*((double*)data.mv_data));
}

Local<Value> valToUInt32(MDB_val &data) {
    return Nan::New<Number>(*((uint32_t*)data.mv_data));
}

[Venemo]

@degifted What I meant by the performance test is: can you create a pull request with a test that I can run on my own machine?