dnote find don't see some documents

[cofirazak]

Environment

• project (cli, server, browser-extension): cli
• version: 0.11.0
• operating system: macOS

Description

dnote find - don't see some documents.

Steps to reproduce

Steps recorded (15 sec) by asciinema: https://asciinema.org/a/307529 Why dnote find - didn't return document with id 65: (65) Basic principles of OOP (Data abstraction) ?

[cofirazak]

i found the reason. the problem is the parenthesis after the word abstraction. if we remove it, or put the space before ) then dnote find adds this document to the result.

[sungwoncho]

@cofirazak thanks for taking time to file an issue with helpful details, and sorry for the confusion.

The behavior seems to come from the way we are building indices for the full text search, as implemented by the following migration:

https://github.com/dnote/dnote/blob/952e0e62f387f6edd026ef199069c1b2a610f463/pkg/cli/migrate/migrations.go#L377

According to https://www.sqlite.org/fts5.html#unicode61_tokenizer, maybe we can treat the parenthesis as a separator, not a token, to change the behavior in cases such as this.

Alternatively, another way to go would be #415, but presently I am not quite sure how to efficiently implement fuzzy search without doing a full table scan. I will leave this issue as open for now for further input.

[malcolmrjones]

I wanted to +1 on this issue. I had a note that didn't appear with dnote find GPL. Note has the content:

# Open Source Licenses
- Apache License
- BSD
- GNU General Public License (GPL)
- MIT License
- Mozilla Public License

It seems not only parenthesis but also curly braces (eg. GNU General Public License {GPL}) and square brackets (eg. GNU General Public License [GPL]) attached to a word evades search by dnote find.