Manage Flavours properly

[kelson42]

Currently each ZIM file is a content, but in fact we have the same content in multiple flavours. We want to be able to:

• search with a flavour as filter
• show in libraries only one content and then choose a flavour (this sould be more readable for users).

[kelson42]

@mgautierfr We have discussed that a few times, but I just remark we don't have any ticket about it. Would be great if you could complete with what it means technically to you.

[mgautierfr]

We have this issue https://github.com/kiwix/kiwix-desktop/issues/316 on kiwix-desktop but it is empty. I may fool myself with the live discussion we had about that but I was thinking I had wrote something about it... Anyway..

The current situation is that every logical zim file is represented by a book in the library. (Several part files (foo.zimaa, foo.zimab, ...) are only one logical zim file) But this representation is a bit too "low level". Several zims files are representing the "same" content but with different variations.

wikipedia_fr_chemistry_maxi_2020-09.zim, wikipedia_fr_chemistry_maxi_2020-10.zim, wikipedia_fr_chemistry_mini_2020-09.zim, wikipedia_fr_chemistry_mini_2020-10.zim, wikipedia_fr_chemistry_nopic_2020-09.zim and wikipedia_fr_chemistry_nopic_2020-10.zim represent all the same content (name) (wikipedia_fr_chemistry), with different flavour (maxi, mini, nopic), and different dates (2020-09, 2020-10)

We can get the name, flavour and date in the library.xml or directly from the zim file in the Metadata. The tag metadata may provide more information about what is in the content (flavour mini has tags wikipedia;_category:wikipedia;_pictures:no;_videos:no;_details:no;_ftindex:yes)

The library should reunite those different files under the same "object" and provide a way to access the exact zim file depending of the requested flavour and date.

I think there is no real technical difficulties here. The real question are more about the functional/api side :

• Should we keep Book as "end user object" and introduce a intermediate object ("Collection", ...) to reunite books ? Or the opposite, use "Book" as a generic object that can access specific flavour/date ("Edition"?)
• Should we handle flavour and date at the same level ?
• Should we go even further and have bigger generic entries for categories ("wikipedia", "wikispecies") and language ?
• How this will be displayed to user ? How it will be queried ?
• Whatever we choose, we must be able to use OPDS catalog (https://specs.opds.io/) to distribute the catalogue.
• We have to think about compatibility. We have reader application that use the opds catalog. We must not break them. We may introduce the new system with https://github.com/kiwix/kiwix-lib/issues/209 (if relevant) and keep root.xml as "plain" library to no break applications.

It could be useful to look the opds spec or what is done on the librarians side to know how they handle this. They probably have the same issue and had more time than us to think about this.

I think that before a implementation, we must agree on a specification of our catalog/library system. Once we have a spec, the implementation should not be too complex to do.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be now be reviewed manually. Thank you for your contributions.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be now be reviewed manually. Thank you for your contributions.

[kelson42]

I start to doubt if we should implement this at this level. Implementing https://github.com/kiwix/libkiwix/issues/545 and providing a way at the UI level to display (per default) only the version of a certain "content-type" might be good enough and would avoid to complexify the libkiwix.

[kelson42]

Superseeded by #545

[mgautierfr]

I disagree with the superseed. #545 is about kiwix-serve main page displaying correctly variant and allowing the user to filter by variant.

This one is far more generalist. It ask the question of how to handle variants. Do not implementing at libkiwix level means that all clients must handle the flavour/variants by themselves and potentially regroup them using different criteria.

[kelson42]

This one is far more generalist. It ask the question of how to handle variants.

This is the job of the UI and for kiwix-serve we want:

• flavour filter
• flavour indicator

I'm not in favour (anymore) of a smart libkiwix being able to detect and merge books together. This would be:

• Heuristical/imperfect
• Complex to code
• Add a new concept in libzim

A sinple filter/labelling is robuster and shoukd be good enough... if not we could the discuss something else.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be now be reviewed manually. Thank you for your contributions.

[rgaudin]

What's the status of this? Now that we're implementing various OPDS client (kiwix-serve, kiwix-desktop, other readers, cardshop), this issue probably deserves a decision.

• All clients wanting to know whether a Book has an update or wanting to unify UI for flavours need that information.
• All clients wanting to store a reference to a Title need this.
• Only ~~identifier~~ we have is the Name (with it's potential conflicts)
• Client can query using name= and then parse/filter the returned entries to build its internal view of a unified Title.

This is simple code that can be duplicated in every client but given our convention is to share stuff like this, I'd prefer this to be settled in the ticket.

Reasons (complexity, maintenance) are legitimate to me and it doesn't prevent us from introducing it at a later stage. This ticket being open but in a Better Library Project brings confusion.