Resolver Not Returning Results, Document AND/NOT/OR queries not possible

[lgtmak]

I am unable to pull results in queries in Resolver .and. I'm using that code:

    const resolveOptions = { 
      enrich: true,
      limit: this.index.store.size,
    };
    const rawQuery = this.index.search({ resolve: false });
    const rawResults = rawQuery.and({ tag: { feedbackScore: ">=4", language: "German" } }).resolve(resolveOptions);

My Document Index config looks like this:

const DocumentIndexConfig = {
  document: {
    id: "id",
    store: true,
    // fuzzy search fields
    index: [
      {
        field: "className",
        tokenize: "forward",
      },
    ],
    // These get tagged as their specific key
    tag: [
      {
        field: "language",
      },
      {
        field: "feedbackScore",
        custom: (data: Training) =>
          Number(data.avgInstructorScore) >= 4 ? ">=4" : false,
      },
    ],
  },
};

I have documents indexed:

{
  “id”: “1234”,
  "className": "My Class Name",
  "language": "German",
  "avgInstructorScore": 4.69,
}

I log rawResults from my code with 0 results.

[lgtmak]

Logging the rawQuery I see this:

W { index: null, result: null, h: 0 }

[lgtmak]

Using the Resolver class yields the same results.

[ts-thomas]

Hey @lgtmak

• Document Resolver is quite limited
• limit: this.index.store.size, might should be removed?
• const rawQuery = this.index.search({ resolve: false }); the query is missing, also document resolver requires to pick one field per query stage

Search by query and also by tag could be simplified. The example you've posted I can't recommend to use resolver. Instead, here a full working example without using resolver:

const index = new Document({
    document: {
        id: "id",
        store: true,
        // fuzzy search fields
        index: [
            {
                field: "className",
                tokenize: "forward",
            },
        ],
        // These get tagged as their specific key
        tag: [
            {
                field: "language",
            },
            {
                field: "feedbackScore",
                custom: (data) => Number(data.avgInstructorScore) >= 4 ? ">=4" : false,
            },
        ],
    },
});


let all_docs = [{
    "id": "1234",
    "className": "My Class Name",
    "language": "German",
    "avgInstructorScore": 4.69,
}];


for (const doc of all_docs) {
    index.add(doc);
}

const results = index.search({
    query: "class",
    tag: {
        feedbackScore: ">=4",
        language: "German"
    }
});

console.log(results);

When just perform a tag search without a query string:

const results = index.search({
    tag: {
        feedbackScore: ">=4",
        language: "German"
    }
});

console.log(results);

[lgtmak]

Your suggestion does not give me an “and” intersection of results. Instead it gives me query fuzzy search results with a union of feedbackScore “OR” language. This is unfortunate as it means I will have to write custom code to intersect the id values then grab them from the store myself. I have had to already write sorting code for returned documents (hence passing the max store size as the limit). However these limitations mean I will probably move to an actual search database that can handle queries such as (written in sql for clarity):

select * from docs where className like ‘%my class%’ AND language = ‘German’ AND feedbackScore = ‘>=4’

Our user won’t always pass a fuzzy query however so sometimes the query will need to be:

select * from docs where language = ‘German’ AND feedbackScore = ‘>=4’

We had need for flexsearch because of it’s speed and ability to import/export on serverless architecture and the small amount of data needed for fuzzy search but it seems we should have evaluated some more mature use cases. It really appeared to simplify not only the above requirements but meant internally I didn't have to deal with the infrastructure team and the 5 people it takes to get a database pipeline setup in modern software development.

[lgtmak]

I was not able to get Resolver working at all for Document indexes even though the code, docs, and commit indicate resolver is available.

[lgtmak]

FYI

I have resorted to writing the following code to solve this issue:

    if (query) {
      const queryResults = this.index.search(query, queryOptions);
      rawResults.push(queryResults.flatMap((r: { result: [] }) => r.result));
    }
    // next get tag results
    const tags = Object.entries(tag ?? {});
    if (tags.length > 0) {
      const tagResults = tags.map(([key, value]) => {
        const results = this.index.search({
          tag: { [key]: value },
          ...queryOptions,
        });
        return results.flatMap((r: { result: [] }) => r.result);
      });
      rawResults.push(...tagResults);
    }
    // then get the intersection of these results
    const [primaryResults, ...remainingResults] = rawResults;
    let intersectionResults: Array<string> = [...primaryResults];
    for (const remainder of remainingResults) {
      intersectionResults = intersectionResults.filter((id) =>
        remainder.includes(id),
      );
    }
    // retrieve documents from store
    const unfilteredResults: EventType[] = intersectionResults.map((id) =>
      this.index.store.get(id),
    );

[ts-thomas]

I've addressed a couple of issues when using resolver on document indexes. Probably related to this: #500 Let me finish fixing them, it should be working afterwards. Document Resolver was one of the latest features and the support is quite limited. Nice to see a higher request of this feature.

[ts-thomas]

@lgtmak Fix was published in the latest vesion. There was some issues with your implementation. The part this.index.search({ resolve: false }) is missing some query or tag.

A full working example:

const DocumentIndexConfig = {
    document: {
        id: "id",
        store: true,
        // fuzzy search fields
        index: [
            {
                field: "className",
                tokenize: "forward",
            },
        ],
        // These get tagged as their specific key
        tag: [
            {
                field: "language",
            },
            {
                field: "feedbackScore",
                custom: (data) =>
                    Number(data.avgInstructorScore) >= 4 ? ">=4" : false,
            },
        ],
    },
};

const index = new Document(DocumentIndexConfig);

let all_docs = [{
    "id": "1234",
    "className": "My Class Name",
    "language": "German",
    "avgInstructorScore": 4.69,
}];

for (const doc of all_docs) {
    index.add(doc);
}

const resolveOptions = {
    enrich: true,
    limit: index.store.size,
};

// a search needs to have "query" or "tag" or both
let resolver = index.search({ query: "class", field: "className", resolve: false });
resolver = resolver.and({ tag: { feedbackScore: ">=4", language: "German" } });
const results = resolver.resolve(resolveOptions);

console.log(results);

You can split resolver stages as you like, e.g.:

let resolver = index.search({ query: "class", field: "className", resolve: false });
resolver = resolver.and({ tag: { feedbackScore: ">=4" } });
resolver = resolver.and({ tag: { language: "German" } });
const results = resolver.resolve(resolveOptions);

The example above could be solved more efficient when combining everything into one query:

const results = index.search({ 
    query: "class",
    field: "className", 
    tag: {
        feedbackScore: ">=4", 
        language: "German" 
    } , 
    enrich: true 
});