ndx
ndx copied to clipboard
localvoid
Feature request - Ability to serialize and load the index
Use case - I have a mobile app with multiple Pouchdb databases and want to implement the content search on that data.
Problem: I want to avoid querying all the databases for their documents and then add those to the document index everytime the app is launched.
I would like to have an API to serialize my index so that I can then persist it to the file system when the app is being closed. When the app is started again next time, I will just load the index instead of querying the databases and creating the index again.
@localvoid I'm considering converting InvertedIndexNode class to an interface since it doesn't have any logic but used as a data container. Are there any reasons for not doing that? Serialization actually works with classes, but right now I don't care about converting plain objects to class instances doing deserialization, so it's working but still not the precise deserialization.
I'm going to use MessagePack serialization format, the serialized data is Uint8Array, arguments:
- On this data file MessagePack works 4-5x times faster than https://github.com/BridgeAR/safe-stable-stringify
- Standard JSON.stringify just fails with
Maximum call stack size exceedederror (it was expected). - Streamed stringifying works, with https://github.com/Faleij/json-stream-stringify for example, but it's slow.
- Serialized MessagePack data file is 20-30% smaller than JSON.
@vladimiry The only reason why I've used classes is to be able to identify this objects in the memory profiler. I don't like how I've implemented everything in this package and I think that it would be better with plain objects and simple functions, so that DCE could eliminate unused code.
Ok, will go with interface then.
I think that it would be better with plain objects and simple functions, so that DCE could eliminate unused code.
One step at a time :)
@localvoid also do you think replacing null with undefined would make sense (optional fields)? In InvertedIndexNode:
next?: InvertedIndexNode<I>;
firstChild?: InvertedIndexNode<I>;
firstPosting?: DocumentPointer<I>;
@vladimiry optimizing compilers like Google Closure Compiler are removing properties with undefined values when object is instantiated, so it will make all callsites that accessing this objects polymorphic. That is why I prefer to use null values.
Just checked latest GCC, and I can't reproduce this behavior on simple examples, so maybe it is ok. I had this problem 4 years ago.
The thing is if I go with MessagePack then it will serialize undefined values as nulls, by design, which would be a sort of inconsistency. This implementation https://github.com/kawanet/msgpack-lite. So I think I better stick to null for now.