Stein icon indicating copy to clipboard operation
Stein copied to clipboard

Implemented more complex seach capabilities using Mango (MongoDB like) query syntax

Open sumartoyo opened this issue 4 years ago • 1 comments

Issue #16 asks for more complex queries.

The solution

Using the solution in this PR, complex queries is written as a PouchDB selector, which is the same as CouchDB selector, which is a Mango query language, which is inspired by MongoDB query language.

Query parsing and rows matching are performed by using pouchdb-selector-core. All comparison operators ($eq, $ne, $gt, $lt, etc.) and logical operators ($and, $or, etc.) have been implemented by pouchdb-selector-core.

The reasonings

Reasonings of using MongoDB query language and pouchdb-selector-core:

  • Current way to write search query still works. Performing search with ?search={column1:value1,column2:value2} still works, and it is the same as ?search={column1:{$eq:value1},column2:{$eq:value2}}. It means it retains backward compatibility.
  • MongoDB is popular, thus it's query language is familiar. In my opinion, for Stein case, it is better to write the query like this than in other query language like SQL or defining our own custom query format.
  • pouchdb-selector-core is used internally by PouchDB and it is well maintained. PouchDB is a mature library and I have not yet met any problems using PouchDB selector. There are another MongoDB query matcher called mongo-parse, but it has security problem (script eval injection).

The implementation

I don't use existing match function in pouchdb-selector-core, which is matchesSelector, because it only perform match for one row. I think it is not efficient to process the query every row. The query doesn't change so it is better to only process the query once and then use it to filter the dataset. This is why I use massageSelector and filterInMemoryFields in the implementation because these two functions is performed matchesSelector.

sumartoyo avatar Jan 18 '21 16:01 sumartoyo

One thing to note for users, query values should always be string or null since data from Stein are always string or null even though it is a number. For example, ?search={"year":{"$eq":2020}} would not work because the value (2020) should be a string ?search={"year":{"$eq":"2020"}}.

Performing to string conversion for all field values is possible but difficult because using logical operators the query object can be deep.

sumartoyo avatar Jan 18 '21 16:01 sumartoyo