kiwix-js
kiwix-js copied to clipboard
kiwix
Libzim experiments
It uses the libzim only for getting content (articles and assets), but without setting the MIME type for now (because the C code does send this information for now) For each content requested, it first resolves redirections without libzim (because it's not implemented in our C code either) All the search features are without libzim for now Error handling is bad when using libzim
It's clearly not optimal but it's a start. It manages to read the content of small ZIM files through libzim. For some reason, it's currently slower than our custom javascript implementation. Maybe because we're copying the data bytes instead of directly reading the buffer.
I have played a bit with this proof-of-concept. First of all, congratulations -- it's a great achievement to have been able to compile libzim with Emscripten and to have overcome the obstacles with stability and the amount of data returned.
While it does "seem" slower than our current implementation, it is not unacceptably slow (at least on a modern PC), and more work can be done to find out where the bottlenecks are. It looks like there could be some positive news coming in the not-too-distant future on #513, and that could well provide faster file access.
This is looking very positive!
(Note to self: link to code for compiling libzim: https://github.com/mossroy/libzim_wasm)
I've rebased this branch, and updated the wasm of libzim, which now allows to use any size of ZIM file, and is based on libzim 7.2.2
Out of curiosity as to whether the libzim port might improve #581, I tried the prototype on an Android phone (with latest Chromium browser). At the moment it does not seem able to do an article count, a search or display an article. I tried with a small Ray Charles and also with two large ZIMs (>4GB). It doesn't seem to be the 4GB issue, as the Ray Charles didn't work either. I didn't connect a debugger to the phone, so I'm not sure what the issue is, but thought I'd document this for further investigation.
Interestingly, I'm getting the same scroll-lock issue when using libzim to read a ZIM with a graphical front page as reported in https://github.com/kiwix/kiwix-js-windows/issues/475. It affects this branch in SW mode, but also jQuery mode (which does not use the libzim WASM).
@Rishabhg71 I've force-pushed this PR in order to get rid of the distracting spacing and layout differences, and also to revert the libzim versions to the same that are in main, so you'll need to reset your local copy to origin. You'll see that, because we already have code to load the libzim WASM for full-text search, actually this PR is very simple, only a few lines intervening in the message channel. For now, I think all you need to do is to make the logic conditional, based on the dropdown you're adding, so that users and developers can easily switch between current implementation and trying out the different libzim versions.
This will require some logic in zimArchive.js, where libzim is currently loaded. You can see the implementation in the PWA, which lets users choose between DEV / ASM / WASM versions of libzim (for the purposes of search, but the loading logic is the same even though your purpose will be to allow reading the contents of a ZIM with the chosen libzim version (as well as search).