Prioritizing/weighting certain FTS fields

[chrisballinger]

When implementing FTS in iBurn, this year I index many more fields in addition to the title, like description, camp name, artist name, etc. I have a global search that searches all objects of all types, in addition to searching on only specific collections.

Because all fields are equally weighted in the results, things that should have a relatively straightforward match based on the title are further down the list than they should be, because many description fields contain similar collections of words.

I'm not sure if this is at all possible given the design of sqlite's FTS support, but it would be great if there was a way to prioritize or set weights to certain columns. Perhaps it would require some higher level trickery in Yap to achieve this.

The idea I had would be to set a YapSearchField object that contains field and weight, instead of setting the string directly:

YapDatabaseFullTextSearchHandler *searchHandler = [YapDatabaseFullTextSearchHandler withObjectBlock:^(NSMutableDictionary *dict, NSString *collection, NSString *key, id object) {
    // current way
    [dict setObject:object.title forKey:@"title"];
    [dict setObject:object.someDescription forKey:@"description"];

    // YapSearchField w/ weights
    YapSearchField *titleField = [[YapSearchField alloc] initWithField:object.title weight:1.0];
    YapSearchField *descField = [[YapSearchField alloc] initWithField:object.someDescription weight:0.5];
    [dict setObject:titleField forKey:@"title"];
    [dict setObject:descField forKey:@"description"];
}];

Could even be backwards compatible with the current behavior by checking for NSString vs YapSearchField.

Whaddya think?

[chrisballinger]

YapSearchField could also be a good place to add a denormalizationBlock

[mickeyl]

This is a great idea. I'd definitely make use of that. Did you have the chance to work on that yet?