[Bug]: Recall incorrect when do vector search with filter when ground truth contains -1

[junqiu-lei]

Describe the bug

We're running a vector search with a filter benchmark, and the recall value is 0.01 when the ground truth neighbors contain -1. The -1 indicates non-existent values. No issues are found when the ground truth doesn't include -1.

To reproduce

PARAMS_FILE configuration for faiss-hnsw-restrictive benchmark:

{
    "target_index_name": "target_index",
    "target_field_name": "target_field",
    "target_index_body": "indices/filters/faiss-index-attributes.json",

    "target_index_primary_shards": 1,
    "target_index_dimension": 128,
    "target_index_space_type": "l2",
    

    "target_index_bulk_size": 100,
    "target_index_bulk_index_data_set_format": "hdf5",
    "target_index_bulk_index_data_set_path": "/home/ec2-user/data/sift-128-euclidean-with-attr.hdf5",
    "target_index_bulk_indexing_clients": 10,
    "target_dataset_filter_attributes": ["color", "taste", "age"],
    
    "target_index_max_num_segments": 1,
    "target_index_force_merge_timeout": 300,

    "query_k": 100,
    "query_body": {
        "docvalue_fields" : ["_id"],
        "stored_fields" : "_none_"
    },
    "filter_type": "efficient",
    "filter_body": {
    "bool":
    {
        "must":
        [
            {
                "range":
                {
                    "age":
                    {
                        "gte": 30,
                        "lte": 60
                    }
                }
            },
            {
                "term":
                {
                    "taste": "bitter"
                }
            },
            {
                "bool":
                {
                    "should":
                    [
                        {
                            "term":
                            {
                                "color": "blue"
                            }
                        },
                        {
                            "term":
                            {
                                "color": "green"
                            }
                        }
                    ]
                }
            }
        ]
    }
    },
    
    

    "query_data_set_format": "hdf5",
    "query_data_set_path":"/home/ec2-user/data/sift-128-euclidean-with-attr.hdf5",
    "neighbors_data_set_path":"/home/ec2-user/data/sift-128-euclidean-with-restrictive-filters-with-neigbors.hdf5",
    "neighbors_data_set_format":"hdf5",
    "query_count": 100
  }

Trigger the run by:

opensearch-benchmark execute-test     --target-hosts $URL:$PORT     --workload-params ${PARAMS_FILE}     --pipeline benchmark-only     --kill-running-processes     --workload-path /home/ec2-user/opensearch-benchmark-workloads/vectorsearch

Get recall result:

|                                                     error rate |         prod-queries |           0 |      % |
|                                                  Mean recall@k |         prod-queries |        0.02 |        |
|                                                  Mean recall@1 |         prod-queries |        0.01 |        |

Expected behavior

Recall should be correct calculated with ground true.

Screenshots

Dataset visualization in sift-128-euclidean-with-restrictive-filters-with-neigbors.hdf5 which contains -1:

Host / Environment

No response

Additional context

No response

Relevant log output

No response

[IanHoang]

@VijayanB Could you take a look at this?

[IanHoang]

Hi @junqiu-lei, how many search clients are being used and what is the number of vCPUs in the load generation host? Depending on the answer, this might be related to this issue that @VijayanB opened up. We're currently looking into recall feature and will provide updates there.

[VijayanB]

I will take a look into this issue