quickwit icon indicating copy to clipboard operation
quickwit copied to clipboard

verify we handle correctly `NOT my-field:my-value` when my-field is tagged

Open trinity-1686a opened this issue 1 year ago • 1 comments

from a quick overfly of postgres metastore, I think we may consider splits unfit if any doc has my-field=my-value, but it should do that only if all docs match (which I'm not sure we can know. if a doc doesn't have my-field at all, it should match that query, but doesn't affect in anyway the tags we store)

we'll have to actually test to make sure there is indeed any problem

trinity-1686a avatar Mar 07 '24 15:03 trinity-1686a

I confirmed that by ingesting a single split with the field 'tagged' tagged, and which takes only the values '1' and '2', searching for * -tagged:1, which should yield exactly the same results as tagged:2, yields nothing. Tag prunning judged the whole split unfit because it contained some documents which have that tag, even though it contains also contains documents with other tags.

This is not a problem with the exact same config if there are instead thousands of tags such that we don't store them in split meta, and we can't do tag pruning

trinity-1686a avatar Mar 09 '24 10:03 trinity-1686a