Same query but different results if executed as NativeQuery vs Fluent API

[warrenbuckley]

Hiya @Shazwazza I have been doing some work with a project using Examine and I have been using the Fluent API to create the query I expect, but its not returning the results I would expect, especially as the lucene query it generates does return results if I use it with NativeQuery() or the Examine Search Dashboard in Umbraco.

Problem

Fluent API and NativeQuery return different results for the same Lucene search query.

Query +__NodeTypeAlias:wviewlibrary +(+wWpPropLibraryAddress:"Scotland" +wWpPropLibraryAddress:"UK")

FluentAPI	NativeQuery
0 Results	3 Results

Fluent API & NativeQuery returning different results

Example Code (Fluent API)

public IEnumerable<IPublishedContent> FilterContent(string nodeTypeAlias, List<Guid> selectedFilters, string locationFilter)
{
    IEnumerable<string> ids = Array.Empty<string>();

    // Examine whats the query of stuff to look for as an array of strings
    // Convert the list of Guids  we have to an array of strings of the GUID IDs
    var guidStrings = selectedFilters.Select(guid => guid.ToString()).ToArray();

    if (_examineManager.TryGetIndex(Constants.UmbracoIndexes.ExternalIndexName, out var index))
    {
        var query = index.Searcher
            .CreateQuery(IndexTypes.Content) // Search only for Content 
            .NodeTypeAlias(nodeTypeAlias); // It must be of nodeType 'wViewCaseStudy' or whatever we pass in as parameter

        // If we have any GUID keys to filter on then add them to the query
        // Otherwise nothing selected so return all of this nodeTypeAlias
        if (guidStrings.Any())
        {
            // GroupedAnd == Try to match content where the Multi Node Tree Picker field contains the picked filter
            // Example field "wWpPropLibraryType" (Contains a CSV of UDI GUIDs)
            // But we are looking in our NEW field we added to the Examine index called "filterIds" - see ExamineContentFilterComponent.cs file
            // which is a space seperated of just the GUIDS without the umb://document/ prefix
            // The code in ExamineContentFilterComponent will always add this NEW field to examine
            // Just need to update the ExamineContentFilterComponent TODOs to ensure it finds other fielters on other document type aliases and properties
            query.And().GroupedAnd(["filterIds"], guidStrings);
        }

        // The user has also entered some form of an address/location/city etc
        // We need to filter the results by this location
        if(string.IsNullOrEmpty(locationFilter) == false)
        {
            // On the Library doctype we know the address is stored in the field "wWpPropLibraryAddress"
            if(nodeTypeAlias == WViewLibrary.ModelTypeAlias)
            {
                // Spliting by comma because the location filter could contain a city and the country
                // For example "Scotland, UK" - so we need to match both
                // With us just searching on the Field() it would match if it found either the city or the country
                // Which would give confusing results
                var locationParts = locationFilter.Split(',');
                var locationPartsExamineValues = new List<IExamineValue>();
                foreach (var part in locationParts)
                {
                    // Calling .Escape() on a string does ExamineValue(Examineness.Escaped, s)
                    locationPartsExamineValues.Add(new ExamineValue(Examineness.Escaped, part.Trim()));
                }
                
                // Commented out this approach as the string parts are not being escaped with quotes
                //query.And().GroupedAnd(["wWpPropLibraryAddress"], locationParts);
                query.And().GroupedAnd(["wWpPropLibraryAddress"], query: locationPartsExamineValues.ToArray());
            }
        }

        // Execute the Examine query and select just the IDs
        // Produces the query
        // +__NodeTypeAlias:wviewlibrary +(+wWpPropLibraryAddress:"Scotland" +wWpPropLibraryAddress:"UK")
        // Which in Examine dashboard works with the results, where via this it doesn't                
        var queryNative = index.Searcher.CreateQuery().NativeQuery("+__NodeTypeAlias:wviewlibrary +(+wWpPropLibraryAddress:\"Scotland\" +wWpPropLibraryAddress:\"UK\")");
        var queryNativeResults = queryNative.Execute();
        var queryNativeResultsCount = queryNativeResults.TotalItemCount;

        var results = query.Execute();
        var totalResultCount = results.TotalItemCount; 
        ids = results.Select(x => x.Id);
    }

    // For all the IDs returned by the Examine query, return the IPublishedContent from the content cache
    foreach (var id in ids)
    {
        yield return _umbracoHelper.Content(id);
    }
}

[Shazwazza]

Well what is the exact string output of the fluent query when you do a ToString() on the query?

Also, could you simplify this to make it as simple as possible to reproduce?

[warrenbuckley]

The fluent query when doing a .ToString() on it returns this "{ Category: content, LuceneQuery: +__NodeTypeAlias:wviewlibrary +(+wWpPropLibraryAddress:\"Scotland\" +wWpPropLibraryAddress:\"UK\") }"

I will update the example and remove my noisy comments and stuff so you have something to work with...

[warrenbuckley]

As I just suck at explaining myself clearly, perhaps this video may help highlight the issue

https://github.com/Shazwazza/Examine/assets/1389894/48cbcb78-82c5-4567-85f6-6f669d5b46df

[warrenbuckley]

Hi @Shazwazza I have written a test to show this issue. Would you want it as a PR submitted to show the issue or just the code pasted here in this issue?

[Shazwazza]

Yeah, if you have written a FluentApiTest as part of Examine codebase, a PR would be the easiest place for me to start.

[warrenbuckley]

One PR which I think highlights the issue https://github.com/Shazwazza/Examine/pull/377

[Shazwazza]

See comment https://github.com/Shazwazza/Examine/pull/377#issuecomment-2037901901