Can't search for C++, #read, other symbols

[0joshuaolson1]

trafficstars

I'm pretty sure #read isn't in the docs yet, but I couldn't tell by searching. Both examples give a huge number of false positives.

[0joshuaolson1]

Is https://github.com/ponylang/pony-tutorial/tree/master/docs/js where a fix would go?

[SeanTAllen]

Not really something we have control over. We host with gitbook whose search was an improvement over read the docs. The symbol lookup chart that gives you phrases to search for was a first step to helping with this perpetual programming documentation issue. Something like that or an index would go a long way.

[SeanTAllen]

https://github.com/ponylang/pony-tutorial/tree/master/docs/js and everything else in docs is deprecated and no longer used.

[0joshuaolson1]

So the only way to fix this is a https://github.com/GitbookIO/gitbook PR, not a configuration change on Pony's end?

[SeanTAllen]

Yes and no: Search is powered by them. Searching programming documentation is often hard in general. So in that sense yes. In the specific case of "#read" not saying ah yes that one thing over there in examples in the appendix, yes, that is very much a gitbook thing.

Addressing with thing like the symbol lookup cheatsheet that tells you what to search for in order to get info on that symbol (and thereby acting a bit like an index) can help.

Finding a way to indicate keywords for chapters or section that search can pick up on would also work.

[SeanTAllen]

Gitbook also makes PDF and Ebook formats available and the search there might work better for this sort of thing.

[0joshuaolson1]

I bothered GitBook's support and repo (since Pony uses the default search backend), but I haven't narrowed it down beyond theories.

[0joshuaolson1]

Even borrow gives no search results, because here it's inside quotes. I asked them again.

[0joshuaolson1]

Someone got back to me:

The search in GitBook is made possible by the plugin-search as you noticed. However the plugin-search [is] powered by a search backend, such as the default one plugin-lunr which uses lunrjs to index the book during the build and to perform queries. This is the plugin used in the Pony Tutorial book.

However, as you can see from the doc of lunrjs, it is tailored by default to process English languages. According to the doc, they trim non-word character by default (so searching for C++ is the same as searching for C). [...] I am going work on improving that. The plugin-lunr itself is quite young, and really simple. Basically the indexing is done here, and the search is made here. So feel free to have a look at it, any suggestion on improving would be appreciated.

He's working on the exact fixes needed!

I have not yet released a new version of the plugin-lunr, but I have fixed several issues. First I added a configuration option for the plugin, which includes by default any special characters (such as in C++). I also fixed the search for some terms (such as "Borrow"), that was caused by escaped HTML.

[Soreine]

Greetings everyone,

I published improvements to the plugin-lunr that powers the plugin-search by default, now in version 1.2.0.

In addition to the fixes mentioned by @0joshuaolson1, I added the possibility to add keywords. To do so, you need to add the list of keywords in the YAML frontmatter of the page in question:

---
search:
    keywords: ["keyword1", "keyword2"]

---

Content of the page [...]

The updated plugin will be part of gitbook by default in the next release of the gitbook library. But you can already use the new version of plugin-lunr by specifying it in your book.json:

{
    "plugins": ["[email protected]"]
}

If you have any feedback or suggestion, feel free to contact me!

[SeanTAllen]

@0joshuaolson1 can you PR the addition of keywords to the various tutorial pages?

[0joshuaolson1]

@SeanTAllen That's a lot of work, and I admit to moving on to another favorite programming language. I'll add lunr 1.2, though (until it's released, then @Soreine said it's unnecessary).

[SeanTAllen]

So at this point, we need someone to take on adding keywords.

[0joshuaolson1]

How is that related to this issue, though?

[SeanTAllen]

Keywords will make it easier to search for various keywords.

[0joshuaolson1]

It's a new GitBook feature, but if search is corrected now, I think the appendix could just list them.

[SeanTAllen]

I'm leaving this open. Keywords would be an excellent addition and make searching more accurate.

[0joshuaolson1]

We could improve the issue title.

[rhagenson]

@SeanTAllen Do you know if the search could be updated as described here since the move away from gitbook?

[SeanTAllen]

Matthias implemented the existing search. I don't know much about it. I think it's lunr based.

[rhagenson]

Since the issue appears to be about site functionality, it seems reasonable to me to mark as a "good first issue" -- looking for someone who can investigate how search works and improve the indexing in general, but especially around less search friendly queries like "#read"

[EpicEric]

FYI, the issue is still present after migrating to Mkdocs.

[rhagenson]

@SeanTAllen I wanted to ping you on this one while I was looking for issues to highlight. As far as I can tell, this issue is "fixed" following improvements to search across upgrades. I do not see any associated docs for keyword search in the material theme and the original symbols are now searchable without issue.