vespa
vespa copied to clipboard
vespa-engine
Special tokens support
As mentioned on gitter, I am trying to query data (with a yql), to find all documents that match the string "&Free". The exact query I'm executing is: "SELECT text_raw FROM sources post WHERE text_raw CONTAINS '&free';" This query only filters documents that contain the string "free", not "&free".
The specific use-case for this is that I am looking for brand names within documents.
Vespa version: 7.165.5
Is the use case here that "&free" is a proper name that you need to match?
Please see match modes https://docs.vespa.ai/documentation/reference/search-definitions-reference.html#match. Default for string fields is text which implies tokenization where punctation characters are removed.
I think this is to make "special tokens" work independently of the linguistics plugin then. I'll try to find some time somewhere to do that.
Yes that would be great if we could use an escape, for example "\x##" to get an exact match of the string. Cheers!
That's no problem but doesn't solve the problem because the engine need to decide at wqrite time what tokens to index.
