Kiwix-serve: suggestion behaviour changed post version kiwix-tools-3.1.2-6-x86_64

[migel-wimtore]

Hi. After recently updating kiwix-tools the default search behaviour supported by kiwix-serve has changed.

Let me try and describe the issue.

Previously it was such that when i searched my Zim file using a web browser served by kiwix-serve, i would be taken straight to a certain page. For instance, when i searched for 'house' i would be taken to the Wikipedia page for 'house', but now i get taken to the results page with many many hits for house and have to trawl through to find desired article.

Is this a regression?

The previous behaviour was really great: it nearly always took me to the page i wanted, and in those cases when it didn't, then i could fall back to a results page.

I have rolled back my install for now (had to install some old libraries) because i really find the new behaviour very unhelpful and clunky.

Looking at the changelog for kiwix-tools it seems that maybe the change 'Use new threadsafe API of kiwix-lib to do suggestions search.' introduced in kiwix-tools 3.1.2 was reverted?

It would be great (if there are those that prefer the new behaviour) if there was a user-facing option to set the search behaviour.

Kind regards and THANK YOU (Kiwix is amazing)

[kelson42]

@migel-wimtore Do you still have this problem with the latest nightly? http://download.kiwix.org/nightly/2022-04-11/

We have fixed a few days ago a major regression and are just about to make a release.

[mgautierfr]

(Moved to libkiwix)

This is a change made with the version 10.0.0 of libkiwix. Issue kiwix/libkiwix#481 (PR /kiwix/libkiwix#497). It seems I missed this change when creating the Changelog. It is wanted as it would prevent user to do a search for house if a house article exist. (You would have to do a house search)

What you could do is to directly try to go to the house article (http://<server>/<bookName>/house), if the house article is not present, you will have a link to do a fulltext search.

[migel-wimtore]

Hi. Thanks both for your responses. Kelson42: i would try the nightly but it seems from mgautierfr's comment that what i was describing was actually intended behaviour -- is this correct?

Mgautierfr: hi, i'm not sure i totally understand your message, but are you saying that the old functionality is not coming back?

[kelson42]

@migel-wimtore Everything is clear now. Not sure what will happen. We have to look/think again at this change in the behaviour of the search and will come back to you.

[migel-wimtore]

@mgautierfr and @kelson42. Thanks for the follow up. As a heavy Kiwix user (no easy internet access) i would like to very strongly express my hope that the old functionality will make a come back!

For one thing Wikipedia itself goes straight to the article (so there is user expectation) but also the search results page organisation isn't the most intuitive: ie there is no hit for 'house' on the first page of results!

When using the old Kiwix version (and also Wikipedia itself) and entering some random terms like 'devo' and 'fish' (sorry for my weird brain) both go straight to the expected pages (for the animal and the band) but neither of those pages appear in the first results pages when using the new version (they are buried and so could just as well not exist)!!!

The cumulative time difference between entering a word, hitting return, and seeing the desired page 95%-plus of the time and having to trawl through pages of (non-semantically organised -- not sure if this is the right term) search results is massive -- you would not want to be doing that very often as it is very time consuming.

Indeed if 'house', 'fish', 'devo', or in fact many things one searches for aren't even on the first few pages of results then common articles are in practical terms being hidden and Kiwix becomes all but unusable!

Thanks for your time and kind regards

[mgautierfr]

There is several things:

First, the behavior you want : It is that you enter term (house) in the search input in the topbar and you press enter, as there is one article named house you go directly to the article. This was indeed a existing behavior and it the behavior of wikipedia.

This behavior was implemented in kiwix-serve by the backend. When you pressed enter in the search input, frontend was doing a search on the house term (http://server/search?content=wikipedia_en&term=house. The search endpoint detect an article was named house and redirect to it (http://server/wikipedia_en/house). It is this implementation that as been removed and will not come back. We want the search endpoint to do a search, all the time.

However the behavior is interesting. The frontend should handle it. If it detect that a article house exists, it should go to this page instead of doing a search. Detecting that the article exists can be done differently:

• Do a request to http://server/wikipedia_en/house and check http return code
• We can extend the result of the suggestion, to mark suggestion with title equal to the search term. If such a suggestion is marked, the front end go directly to it
• We could change a bit the behavior of the suggestion system to always validate the first entry of the suggestions list on enter event and so, always go to the "closest" article. If the user want to do a search, he has to explicitly select the last entry "De a search with <term>..."

[migel-wimtore]

@mgautierfr. Hi. Yes i want very much the behaviour you describe in the first paragraph of your most recent post.

Very, very much.

Actually i struggle to imagine that anyone asked would /not/ want it. As this is the way Wikipedia works and also it /just/ worked AKA was a good user experience.

As i also mentioned the search results i am seeing for, for example 'house' (and those other things i mentioned), don't even have a corresponding hit appearing in the first few pages of results -- and paging through results looking for what you might reasonably expect would actually be an 'exact hit' (i.e. the article for 'house' when i typed 'house') is a pretty unenjoyable and frustrating task!

I imagine we all agree on this.

It's almost a shame that the behaviour, as was, was removed at all. Frankly it worked and having to suffer the absence of what was a good feature for the sake of 'code cleanliness' or 'pure design principles' (please forgive me if this is an incorrect assumption on my part) is something i think practically all users would reject given the choice.

Thank you so much for your time and for explaining everything to me.

Best regards

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be now be reviewed manually. Thank you for your contributions.

[mgautierfr]

@juuz0 It may be interesting to have this in mind as you are working on the kiwix-serve frontend

[juuz0]

@mgautierfr @kelson42 imo, this would be the better one:

• We could change a bit the behavior of the suggestion system to always validate the first entry of the suggestions list on enter event and so, always go to the "closest" article. If the user want to do a search, he has to explicitly select the last entry "De a search with ..."

[juuz0]

@kelson42 Opinion on this ^? I will be on it right after noJquery PR is merged

[kelson42]

We discuss here only the behaviour of kiwix-serve title based suggestion system when user press ENTER without selecting a suggestion.

I'm not in favour of always going directly to the first suggestion, because in many scenario it won't fit good enough the search pattern. But I understand that we have lost here a bit of usability.

I think we should go directly to the first suggestion only if this suggestion is almost what has been typed in the textbox area. Otherwise we should still start a fulltext search.

There is many approaches possible, but the best would be to have the backend sharing the pertinence of the result in the json. Then, if good enough (bigger than 90% for example), the js would pick it.

@juuz0 Does the Xapian delivers such a matching note for each suggestion?

[kelson42]

@mgautierfr @veloman-yunkan Only based on Xapian suggestion list, can we ask today the libzim to return one article which fully match (modulo diacrirics, case, ...) the search pattern? Or would that need the implementation of a new primitive "findExactTitle()"?

[mgautierfr]

Making libzim doing it is difficult. zim suggestion api (and all other search/find api) returns a iterator which generate the result on the fly. So it would be difficult to insert a specific entry in front of the iterator.

On top of that, changing the libzim behavior is arguable:

• Cons: If suggestion search is xapian search, we should returns xapian search only.
• Pros: If suggestion search is suggestion, we should returns suggestions, whatever they come from.

Adding this on libkiwix should be easy however. We are feeling a vector from the result of the iterator, we could add a entry in the vector first easily.

Or would that need the implementation of a new primitive "findExactTitle()"?

This already exists and it is Archive::getEntryByTitle

[kelson42]

Or would that need the implementation of a new primitive "findExactTitle()"?

This already exists and it is Archive::getEntryByTitle

@mgautierfr AFAIK getEntryByTitle() does not "returns one article which fully match (modulo diacrirics, case, ...)" and the documentation does not tell that. You confirm?