Package Kiwix JS or Kiwix JS Windows as a portable Electron app

[Jaifroid]

For fun, I tried shoehorning Kiwix JS Windows into the basic electron-quick-start app, and it launched with no problems. See screenshot below. It set me thinking that this could be a good way to provide Kiwix JS on a thumb-drive, which is something @Popolechien has been asking about for quite some time. Since Electron apps have access to the filesystem, it should be possible to enable Kiwix JS as a packaged app for this kind of purpose, e.g. with WikiMed or Wikivoyage. Could this be interesting @mossroy @kelson42 ?

[Jaifroid]

FYI The branch with the basic code is https://github.com/kiwix/kiwix-js-windows/tree/electron . Instructions to build:

• Clone the repository and switch to electron branch
• cd to kiwix-js-windows
• npm install
• npm start

NB cookies are not currently remembered between sessions, so I guess another Electron method would need to be used to store settings.

[kelson42]

To me such an app would be primary interesting if it allows us to reach users who would have otherwise problems to read ZIM files. Not sure we have a lot of them currently.

[Popolechien]

@kelson42 I do see a couple of opportunities for this so wouldn't discount it altogether. Also, tbh, kiwix-desktop 2.0 isn't exactly working yet so let's keep our options open. @Jaifroid do you think there would be a size limitation in such packages? Or to put it more clearly, could we have a one-click Wikipedia package with all 5 million articles (and 78Gb of it)? I doubt electron was thought for this, but hey, why not.

[kelson42]

@Popolechien Can you elaborate about the needs?

[Popolechien]

We've had requests in the past of people wanting to distribute Wikipedia (German) as a goodie on branded flashdrive. They were put off by the complexity of having to open the folder, find the relevant .exe, open the zim file. A one-click solution would be a neat thing. Other use case: flashdrives for freedom, in NKorea. They're only interested in one content (krwp), so that would be a huge gain in UX.

[kelson42]

@Popolechien Making a flashddrive with Kiwix Desktop 2.0 Beta5 is possible. The question of the distribution on our side is a different problem, we have stop to make it because the added value was almost null (since ft indexes are int he ZIM) and because this consumes a lot of hardware resources. I'm not aware about any ticket open regarding the needs given in the above comment. It's important to open them to then discuss about the technical solution (and not the other way around). @Jaifroid I don't see any advantage in comparison to our current Kiwix-Desktop solution. Do you?

[Jaifroid]

Hope you're all surviving the heat! Cambridge was the hottest place in the UK today... I'm interested in experimenting to see how easy or not making a packaged app would be. Since JS in the ZIM works in Service Worker mode, and Electron supports Service Worker, we could have quite a good solution for supporting a wide range of ZIMs on a flash drive.

Currently, Kiwix Desktop Beta feels slower to launch and load pages than the test Electron Kiwix JS, app, and Desktop doesn't support ZIMs with subdirectories. OTOH, Kiwix JS doesn't yet support full text indices. That's not so necessary for Wikipedia, however.

Anyway, there's no harm in experimenting a bit, and then we can see what's feasible and desirable.

[kelson42]

Currently, Kiwix Desktop Beta feels slower to launch and load pages than the test Electron Kiwix JS

It is true that at the launch time it seems slow to me too... I will open a ticket about that.

Desktop doesn't support ZIMs with subdirectories.

I don't see any bug open about that. Could you please explain here what fails?

[Jaifroid]

@Jaifroid do you think there would be a size limitation in such packages? Or to put it more clearly, could we have a one-click Wikipedia package with all 5 million articles (and 78Gb of it)? I doubt electron was thought for this, but hey, why not.

@Popolechien I don't know for sure yet. File size isn't a limitation in Kiwix JS, or in the test app, but I haven't yet tried file access with Electron APIs (i.e. without a file picker), which is the main requirement of a 'one-click' solution. Will report back.

[Jaifroid]

Desktop doesn't support ZIMs with subdirectories.

I don't see any bug open about that. Could you please explain here what fails?

@kelson42 I opened https://github.com/kiwix/kiwix-desktop/issues/206 on 15th July. I've just added info on how to reproduce the issue specifically with Stackexchange ZIMs.

[Jaifroid]

I had a bit of fun this weekend working out how to launch a ZIM file automatically without picking it in the Electron app. So, please find a demo of "Wikivoyage by Kiwix Electron Edition" here (I'm using Wikivoyage because it is smaller than Wikimed for testing purposes):

https://github.com/kiwix/kiwix-js-windows/releases/tag/v0.9.9.92-Wikivoyage-Electron

For full details, see the release notes on that page, but basically it's a portable app for Windows. It just needs to be downloaded and unzipped anywhere, e.g. on a thumb drive. If unzipping to a thumb drive, for ease of use it would be sufficient to unzip the contents of the top-level folder (and all sub-folders) so that the executable kiwix-js-windows.exe is in the root of the drive.

@Popolechien , I think this fulfils your requirements for a one-click (well, double-click) solution.

Mac and Linux packages could be built easily, but I can't currently test for those platforms (the Windows Subsystem for Linux cannot run Chrome with an X-Server).

Any ZIM archive could be packaged with this app, but see the Release Notes on the download page for details of some adjustments to the initialization file to autoloaunch a different ZIM. I've tested dropping in the full 80GB English Wikipedia, adjusting the file to load in init.js, and it works fine! The ZIM doesn't need to be part of the distribution package (zip file): it can be dropped in later.

Note there is no access to the full text index (this functionality has not been implemented in Kiwix JS), though it's not so necessary for Wikipedia and Wikivoyage.

If you don't want to download the binary, you can run with Electron like this:

• Clone kiwix-js-windows and switch to electron branch
• cd to kiwix-js-windows
• Copy the latest Wikivoyage ZIM into the archives folder and rename to wikivoyage_en.zim
• npm install
• npm start

[Jaifroid]

@mossroy Some technical details which may be interesting. I used Kiwix-JS-Windows because it has ready-made entry points for a packaged app, but I think the code can easily work with Kiwix JS. Main things are:

• Use Electron APIs for reading the file instead of FileReader. Fortunately we only use FileReader in one place, in util.js, and the main change is here;
• Construct a fake File object, because Electron doesn't provide a way of converting its ArrayBuffer into a File object;
• Import the required Electron APIs into the DOM, because we can't use Electron's require as well as our own require.
• As explained in the Release Notes, we can either have document.cookie or Service Worker, but not both. However, there is a custom Electron API for cookies, so we could override our current cookie setter and solve this issue. For now, the app runs in jQuery mode only.

[Popolechien]

@Jaifroid Thank you, am currently downloading and will report back, but here's a first obvious question: the zip file is named kiwix-js-windows-win32-ia32.zip Am I right to assume it should run on 32bit computers?

[Jaifroid]

@Popolechien It runs on 64 bit for sure (my machine is 64), and I assume from the name that it runs on 32 bit, but haven't tested.

[Popolechien]

Ok, I've asked for help, let's see if anyone has an old Windows XP machine lying around: the best use case for that would be African deployments, as a couple of people have been badgering us for "abandoning" 32bit-machines with Kiwix-desktop 2.0.

Other than that, the app looks like it's working fine!

[Jaifroid]

@Popolechien Whether it runs on XP will depend entirely on whether the Electron framework supports XP (and perhaps whether Chromium webview has dependencies). We can probably find that out.

[Jaifroid]

@Popolechien It supports Win7 32bit, but not XP according to https://electronjs.org/docs/tutorial/support#windows . But with XP now a security risk and Win7 reaching end-of-life soon, that's not too shabby. The best bet for the African deployments is probably to use Kiwix JS with Firefox or Chrome, assuming one or other browser in a decent version is still supported on XP.

[Popolechien]

Yep, Windows XP SP3, doesn't work. Entrypoint procedure AcquireSRWLockExclusive cannot be found in dynamic linking library KERNEL32.dll. Well that's too bad. Also and more generally speaking, if anyone is still stuck with WinXP/Vista/7, security probably isn't their main concern. But let's stick to the Kiwix-JS extension for that particular scenario.

[Jaifroid]

OK, thanks for the test. Yes, Kiwix JS in extension format would be best for such users, assuming the they have a supported version of Chrome or Firefox.

[mossroy]

For Firefox, version 52.9 ESR is the last version that supports Windows XP. See https://support.mozilla.org/en-US/kb/end-support-windows-xp-and-vista This version is still supported by Kiwix-JS. We have unit and UI tests running on Firefox 45 : even if they are unstable (see https://github.com/kiwix/kiwix-js/issues/536), I regularly test on Firefox OS (which relies on Gecko 44).

For Chrome, version 49 is the last version that supports Windows XP. See https://chrome.googleblog.com/2015/11/updates-to-chrome-platform-support.html and https://en.wikipedia.org/wiki/Google_Chrome_version_history I'm not sure if this version is supported by Kiwix-JS. I'd say it's probable, but our unit and UI tests currently run on Chrome 58 (plus latest Chrome).

[Jaifroid]

I've improved the app considerably, and have presented it as a generic app packaged with wikipedia_en_100.zim (i.e., this version does not have Wikivoyage, so is a much lighter download):

https://github.com/kiwix/kiwix-js-windows/releases/tag/v0.9.9.92-Electron

Use case:

• The binary is 32-bit, so it will run on Win 7 32-bit and up (of course also on Win 64-bit), unlike Kiwix Desktop Beta which I understand requires a 64-bit OS;
• It could be used on a thumb-drive for Windows machines, but can easily be compiled for Mac and Linux, and I guess all three could be on a single thumb-drive.

The main improvements:

• The app can now remember a picked ZIM (anywhere on the user's system) and will load the last picked ZIM automatically on next app launch (the user does not need to pick the archive again);
• The app will launch the last-visited page on startup (this can be turned off for privacy reasons);
• Boot loop protection added in case the picked ZIM (or the packaged ZIM) no longer exists;
• Very fast start-up;
• Improvements to URL rewriting;
• Several things causing the app to hang on startup under some conditions have been ironed out.

NB Service Worker mode is still not available in this version.

https://github.com/kiwix/kiwix-js-windows/releases/tag/v0.9.9.92-Electron

[Popolechien]

It could be used on a thumb-drive for Windows machines

Nice, but when I earlier referred to a single file service I really meant having only Kiwix.exe on my desktop (I currently have a folder with 15 files and 4 sub-folders).

The app can now remember a picked ZIM

Very nice!

[Jaifroid]

Nice, but when I earlier referred to a single file service I really meant having only Kiwix.exe on my desktop (I currently have a folder with 15 files and 4 sub-folders).

Hmm. That would be a bit tough, as we'd need some kind of filesystem encapsulated inside an exe, with the code to read the encapsulated filesystem. And if we used an unpacking approach, then anything that unpacks the FS at runtime would have to contend with potentially unpacking several gigabytes of a ZIM file, which would make for very slow startup time.

Practically, I think the best we could do would be to have a portable shortcut (batch file) and a folder. The shortcut would run the .exe inside the folder (it can be included in the zip). It would look fairly neat from the end-user's point of view, and wouldn't "pollute" the thumb drive too much. See screenshot. Batch file could be called anything, like Start Kiwix JS.bat.

[Jaifroid]

@Popolechien , I've done a bit of investigation, and it turns out it is possible to build the app as a single standalone portable exe, using the electron-builder npm package. I've built an exe and can upload it if you're interested in testing, but IMHO it has disadvantages that outweigh the convenience of having everything in a single .exe:

• The app is much slower to launch. If it is just the app with no packaged archive, it takes about ten seconds to launch on a good machine, versus one or two seconds for the current version. If it has a large packaged archive, it takes double that;
• There's no magic involved: in the background, the app automatically unpacks a 7zip archive into local temporary folders to run, hence the longer startup time and overhead;
• Given this, it's only practical to distribute the app without an internally packaged ZIM, and this means the end user would have to pick a separate archive on the thumb drive;
• All of this is IMHO less convenient than providing a thumb drive with a batch file and a folder as illustrated in screenshot above. In that configuration, the app loads very fast and the user does not have to pick the startup ZIM.

[Popolechien]

@Jaifroid Ok, and then what if we take the archive out altogether (at least from the opening sequence)? What about opening time then?

[Jaifroid]

@Popolechien , I've uploaded a version with no packaged archive at all, so you can test for yourself:

https://github.com/kiwix/kiwix-js-windows/releases/tag/v0.9.9.92-Electron-Standalone

Note that I've zipped it to prevent problems with downloading a bare exe. Additionally, when you first launch the exe, Windows SmartScreen filter will flag it as unsafe (because this dev version is unsigned). This is expected behaviour, and it's safe to continue. Obviously, a released version would be signed.

Of course you'll need to pick your own ZIM file as none is provided. However, once picked, the file should be remembered on next launch. See release notes for further details, but I'd just flag that it works best with Wikimedia ZIMs. If you want to try a ZIM with dynamic content, go to Config and switch to Service Worker mode, but it's still a bit rough in that mode.

[mossroy]

Thanks a lot for these investigations @Jaifroid : it looks very interesting. It's great that the Electron APIs allow us to remember and automatically reopen the ZIM file : it's a popular demand that we unfortunately can not implement within a browser extension (at least with the APIs Firefox and Chrome currently provide) Having the SW mode would also be a workaround for the fact that they are currently disabled in Firefox extensions. One thing we'll miss (compared to extensions) is the deployment/feedback infrastructure provided by the stores. NB : I have a very limited Internet access for now, and could not test myself.

[Jaifroid]

Having the SW mode would also be a workaround for the fact that they are currently disabled in Firefox extensions.

I've enabled SW mode (in the latest pre-release) and have compensated for lack of cookie support by patching our cookies script to use localStorage (when running under Electron). The API for localStorage is almost exactly the same as for the cookies script, so it's a very easy patch.

[Jaifroid]

I've made a new pre-release of the packaged app:

https://github.com/kiwix/kiwix-js-windows/releases/tag/v0.9.9.93-Electron

Service Worker mode is fully functional and has near feature-parity with jQuery mode. JQuery mode (the default) is still quite a lot faster because it does not extract or use scripts from the ZIM (and scripts are not cached, unlike CSS). When a ZIM needs SW mode because it has dynamic content, the user is warned to switch with a banner, with a "show-me" style link (this is a feature of Kiwix JS).

[Popolechien]

Ok FWIW someone tried it on Windows XP SP3 32-bits and unfortunately it does not work :-/ (but >= Vista does, apparently)