qdrant icon indicating copy to clipboard operation
qdrant copied to clipboard

Support for array values in FieldCondition

Open williambrach opened this issue 2 years ago • 0 comments

Hi! I discovered a potential new feature for the qdrant Filter capabilities.

Problem

In our qdrant instance, we score our custom implementation of the ID (wineId) in the point payload. For example, here's the payload structure:

payload={'adventurous': 0, 'companyId': 5, 'traditional': 1, 'wineId': 'PSOMM-36129206206421651802760151071810121225'}

It's important for our use case to be able to delete qdrant points based on the "wineId". However, I encountered a bottleneck when attempting to implement a bulk delete of these points using our ID. There isn't a way to insert multiple IDs into a single Filter clause.

After a discussion on Discord , I came up with a solution:

conditions = []
# create conditions
   for wineId in d['wineId'].values:
       conditions.append(
           Filter(
                   must=[
                       FieldCondition(
                           key="wineId",
                           match=MatchValue(value=wineId),
                       ),
                       FieldCondition(
                           key="companyId",
                           match=MatchValue(value=companyId),
                       ),

                   ],
               )
       )
# run qdrant delete        
   QDRANT_CLIENT.delete(
       collection_name=f"{collectionName}",
       points_selector=FilterSelector(
           filter=Filter(
               should=conditions
           )
       )
   )

In this solution, I generate an array of conditions with all the "wineIds", and then call a Filter with the "should" clause on all these conditions. This solution performs well for us, as it deletes 14k points in approximately 3.5 seconds.

Proposed Solution

My proposal for a new Filter addition is to support the "in / isin" operator, similar to what is supported in MongoDB in. For example, here's how it could be used with the qdrant Python client:

    QDRANT_CLIENT.delete(
        collection_name=f"{collectionName}",
        points_selector=FilterSelector(
            filter=Filter(
                 must=[
                        FieldCondition(
                            key="wineId",
                         # proposed solution   
                        **match=MatchValueIn(value=wineIds)**
                        ),
                        FieldCondition(
                            key="companyId",
                            match=MatchValue(value=companyId),
                        ),

                    ],
                )
            )
        )
    )

I understand that this proposal is more focused on improving the code readability, but it would greatly enhance the ease of use and convenience for users.

williambrach avatar Feb 03 '23 15:02 williambrach