mwoffliner
mwoffliner copied to clipboard
openzim
Running mwoffliner on WSL
Installed all the dependencies, still getting this build error.
using Ubuntu 20.04 on WSL 2 (installed on Windows 11 Pro). Also, tried using installing Ubuntu 20.04 on a remote EC2 instance, getting the same error, so maybe it's not because of WSL.
Find the complete logs attached
2022-02-12T07_57_19_753Z-debug.log
I don't understand what you do. Can you share the full log (with the command you run) please?
BTW, a few months ago, I tried to get it work with WSL myself, without success. I recommend to better install an Ubuntu in a VM (like with virtual-desktop).
Find the complete logs attached
You can find the complete logs attached here. I ran the following command inside the project folder:
npm ci
after cloning the repo. I'm using the following versions of node and npm:
node v14.19.0 npm v6.14.16
Ok, then this is a bug. Like I said, we tried with @Jaifroid as well and failed too.
So, should I try with Virtual box only? I have an Ubuntu machine on my AWS ec2 instance as well, getting some error there as well, should I explore that further or go with the virtual box? Actually, I tried the virtual box thing before but it hangs a lot that way due to lack of graphic acceleration I guess.
Ah yes, I remember! For us it was a severe dependency problem, and didn't work on WSL Ubuntu. It might be worth seeing if the dependency issues also occur on Hyper-V or Virtual Box. Ubuntu runs fine on Hyper-V (using a Generation 2 machine, 2 virtual processors, install it from a mounted ISO into 64GB VHDX). It may be better supported on Windows 11, though I don't know specifically about graphics performance -- it'll never be great virtualized. I'm sure Virtual Box is fine.
okay, thanks. Will try using a virtual box. But, I think we can work on the bug for making it work with WSL as well. As many developers today are adopting that workflow. I can work on that if we can identify the exact problem.
But, I think we can work on the bug for making it work with WSL as well.
That would be brilliant, if possible. I agree WSL Linux distros are becoming very popular because it gives near-native performance at least with its own FS (less so if accessing lots of files on one of the ntfs mountpoints like /mnt/c/, which can be slow with big npm install operations).
@gaurav7019 It would be indeed great if we find a way to get MWoffliner working on WSL.
I'll try exploring this alongside, let me know if you've any leads already that can help.
@juuz0 said
I was able to install it correctly (except some deprecated warnings) on my WSL (Ubuntu 20.04, Windows 11). Commands I followed:
sudo apt update sudo apt install redis-server sudo apt install libjpeg-dev autoconf automake gcc sudo apt install nodejs sudo apt install npm git clone https://github.com/openzim/mwoffliner.git cd mwoffliner sudo npm i -g mwoffliner (this failed) sudo npm cache clean --force (important) sudo npm i -g mwoffliner --unsafe-perm mwoffliner --help (yay!)
I'm still getting the same error while installing the dependencies using npm ci. I am building the code repo for development purposes.
@juuz0 Here, BTW, what's the need of cloning the repo if you're installing the package globally anyway using npm?
@gaurav7019 I had some modules missing error, so I changed to cloned directory but yeah you might not need to. I'm not sure :)
Also, it fails at libzim step (did for me too), so have you tried forcefully cleaning cache and then
sudo npm i -g mwoffliner --unsafe-perm
(sudo and --unsafe-perm are important)
This isn't a WSL problem tho, it is probably just a permission issue, which should go away after force cleaning cache and re installing
@juuz0 I tried those steps, and got a long way, but it fell over again on sudo npm i -g mwoffliner --unsafe-perm. Screenshot showing commands from git clone onwards. This is Ubuntu 20.04.3 LTS on Windows 11 running in terminal.
So I suppose ssh access is somehow not configured, even though git can clone the repo fine.
We have a few dependence repos available via SSH and related to the umaintained Wikimedia packages.This is causing more and more problems looks like, see for example https://github.com/openzim/mwoffliner/issues/1592 open yesterday.
I have no real easy solution, except setup the client ssh infrastructure in WSL (shoukd be doable). Middle term, we should remove all this code (which is responsible for running a local Parsoid in case the upstream backend id old. This as well allow to have mobile output if not configured upstream).
Might help https://simplernerd.com/git-ssh-keys/
I had that error too when I tried on my main Ubuntu (but not on WSL, must be because ssh was already configured with GitHub?) . I followed this: https://docs.github.com/en/authentication/connecting-to-github-with-ssh/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent
Related: https://docs.github.com/en/authentication/troubleshooting-ssh/error-permission-denied-publickey
@kelson42 That link is nice, should add in README after installation succeeds for @Jaifroid @gaurav7019
@juuz0 If we have a procedure which works for all, the yes we should update the README with WSL instructions
I've managed to get past the ssh error by setting up keypair and adding to GitHub (warning to anyone following this, do not use a passphrase with the key because the npm script can't cope with that). But now a new error:
By investigating #1592, I had such an issue yesterday on GNU/Linux. I believe this is tied to "-g" part of the command. Maybe this ticket might help to understand better what is going on https://github.com/npm/cli/issues/624.
OK, thanks. The only solution I could see there was not to run this as sudo. If I don't run as sudo, then that particular error goes away, but then I get later permission errors, e.g. for EACCES.
Only other option I could see in that thread was to move the clone of the repo out of user directory and start again. I'll try that later, though I have my doubts because this install isn't supposed to depend on the cloned Repo as @gaurav7019 mentioned. Presumably it updates the global npm config, though I don't know if that is also per-user.
Using --unsafe-perm flag with sudo npm i -g mwoffliner works well for me on WSL (without cloning the repo as I am installing the package globally). But, for dev purposes, cloning the repo and building it inside it's folder using nom ci --unsafe-perm throws a build error attached below.
@Jaifroid Please confirm if you're able to reproduce this. So, we can move forward in this direction.
Using
--unsafe-permflag withsudo npm i -g mwofflinerworks well for me on WSL
It didn't work earlier because I was using nvm for nodejs and npm. So, sudo was not working with npm and node. Fixed that to make it work fine.
As I mentioned earlier, I spinned up a remote Ubuntu machine on AWS, SSHed into it from my vscode and tried building the repo there using npm ci. And, I got the same build error as mentioned above. So, I wonder, is it really related to WSL or are we missing something here.
https://stackoverflow.com/a/47143457
As per this answer, this error could arise from resource limitations. Both my AWS instance (free tier, so fewer resources) and WSL might be having this. So, maybe this is the right direction. I am going to explore this for now.
@gaurav7019 @juuz0 @Jaifroid Do we have a complete and robust set of commands to achieve to get a local version (from git) of mwoffliner working on WSL?
Same question with global install from npmjs.org?
I need to check, I'm currently on a different machine. Will get back to you.
No, I'm still getting the same "could not create leading directories" error. And I realized that I was already using --unsafe-perm. @gaurav7019 are you saying that you can install with that or that you can't? Your screenshot showed a lot of errors, but you say it "worked well". In any case, for me it doesn't work, unfortunately. I'm not using nvm and I'm on the latest node in the Ubuntu repo.
@Jaifroid I was able to successfully install the package globally using npm.
Those errors are when I try to install it locally using npm ci --unsafe-perm inside the code repo folder.
So you never saw the permission errors above about not being able to create directories? I don't think it's a resources error, it's clearly a permissions error., probably a conflict between git expecting a user and root not being a normal user. Could you tell me which versions of node and npm worked for you? I am also trying to install globally.