Asar support

[HZSamir]

Hello, First off I'd like to thank you for sharing this, however this repo still hasn't solve the single biggest problem of Meteor/Electron apps on Windows, that is Asar support. As you probably know, these projects usually have a great number of files and nested folders, hitting quite easily the 260 characters limit on Windows, and to this day, I haven't been able to reconcile it with Asar.

I am sure you have encountered this issue as well and I would like to ask you for your take on it. Is is possible to package the app inside an asar file or should I give up on this? In a word, what is your opinion?

Thank you.

[rsercano]

Hello @Unforgiven-wanda

you're definitely right, I'm currently working on it :)

[HZSamir]

@rsercano Good to know. Please let us know when you make progress :)

[rsercano]

Hi @Unforgiven-wanda

I've managed to unpack binaries (node and mongod) to outside of asar package and managed to spawn mongod successfully, but no luck to spawn node due to limitations of electron asar packaging.

Tried fork, execFile and a few other variations but still no luck, do you have an idea or what's your progress on this, maybe it's time to combine our strengths :)

fork fails with

Error: ENOENT: no such file or directory, uv_chdir
    at Object.<anonymous> (D:\git\mongoclient_packaged\packaged\Mongoclient-win32-x64\resources\app.asar\app\main.js:8:9)
    at Object.<anonymous> (D:\git\mongoclient_packaged\packaged\Mongoclient-win32-x64\resources\app.asar\app\main.js:10:3)

execFile fails with Cannot find module error

Probably fork cant find dependent modules in relative path of asar package which causes more trouble.

[HZSamir]

Okay, so I'm on a business trip right now, I don't have my files in front of me, but just off the top of my head: I extracted all executables using electron-builder asarUnpack option, and changed all spawn calls in my main file to execFile. Other than that, running meteor's main.js inside the Asar file results in all the paths being jumbled (something like requiring my/path/serverJson.js instead of my/path/app.asar/serverJson.js. To fix that, I found this nifty code to monkey-path the fs module here (You can read more about it here)

So, in the main.js file of meteor, you'd get something like:

const path = require('path');
process.chdir = (newPath) => {
  cwd = newPath;
};

const originalReadFileSync = require('fs').readFileSync;

require('fs').readFileSync = (filePath, ...args) => {
  let newPath = filePath;

  if(specialFiles.includes(filePath)) {
    newPath = path.join(cwd, filePath);
  }

  return originalReadFileSync(newPath, ...args);
};

//Meteor starts here

I also recall something about the web.browser paths being jumbled anyway, so I had to fix them manually inside Meteor's files. But here's the thing, for all those fixes, at the end I encountered an incomprehensible error, that I just couldn't fix. I'll post here again first thing when I get back.

I hope this helps you some. Please keep me informed of your progress :)

[rsercano]

Thank you so much for sharing, definitely will do :)

[HZSamir]

Hello again, So here's how I did it. First I Demeteorize the app (see this) basically just run the build command. Then in the main.js file (I merged both Meteor's main and the main file of my Electron project):

// The debugger pauses here when you run `meteor debug`, because this is 
// the very first code to be executed by the server process. If you have 
// not already added any `debugger` statements to your code, feel free to 
// do so now, wait for the server to restart, then reload this page and 
// click the |▶ button to continue.

process.env.MONGO_URL='url/to/my/mongo';
process.env.ROOT_URL='http://localhost:8080';
process.env.PORT=8080;

// Monkey-patch the chdir method to work inside Asar
const path = require('path');
let cwd = '';

process.chdir = (newPath) => {
  cwd = newPath;
};

const originalReadFileSync = require('fs').readFileSync;

require('fs').readFileSync = (filePath, ...args) => {
  let newPath = filePath;
  return originalReadFileSync(newPath, ...args);
};

//Notice I added app.asar to the paths
process.argv.splice(2, 0, 'resources/app.asar/programs/server/program.json');
process.chdir(require('path').join(__dirname, 'programs', 'server'));
require('./programs/server/boot.js');

const electron = require('electron')
const app = electron.app
const BrowserWindow = electron.BrowserWindow

const url = require('url')
let mainWindow

function createWindow () {
  mainWindow = new BrowserWindow({width: 800, height: 600})
  mainWindow.loadURL('http://localhost:8080')
  mainWindow.webContents.openDevTools()
  mainWindow.on('closed', function () {
    mainWindow = null
  })
}

app.on('window-all-closed', function () {
  // On OS X it is common for applications and their menu bar
  // to stay active until the user quits explicitly with Cmd + Q
  if (process.platform !== 'darwin') {
    app.quit()
  }
})

app.on('activate', function () {
  if (mainWindow === null) {
    createWindow()
  }
})

boot.js file:

//Replace this line
var serverJsonPath = path.resolve(process.argv[1]);

server-json.js file:

var serverJsonPath = path.resolve(process.argv[1]);

boilerplate-generator.js file:

// Fix the path for head.html
var readUtf8FileSync = function (filename) {
  filename = path.join(path.resolve('.'), 'programs', 'web.browser', path.basename(filename));
  return Meteor.wrapAsync(fs.readFile)(filename, 'utf8');
};

That's about it. here I encountered some error I couldn't fix, got bored trying to fix it for a few hours before I gave up.

Hope this helps.

[rsercano]

Thanks for sharing @Unforgiven-wanda as soon as I get some spare time, gotta check that.

[rsercano]

Hi @Unforgiven-wanda

I'm trying to use execFile instead of spawn for the node process but it fails to find main.js wit the error Cannot find module

How did you manage to use it ?

p.s. fork finds main.js file but I don't want to rely on the node process of the client.

[HZSamir]

Usually you just require it:

var child_process = require('child_process');
var myScript = require('path/app.asar/main.js');
// Then you execute it
child_process.execFile('node', [myScript], function(error, stdout, stderr){
	console.log(stdout);
});

[rsercano]

Unfortunately that didn't work for me, I'm at work now, can share my work tonight.

[rsercano]

Ok, here's what I did:

 let nodeProcess = execFile(path, [mainPath], (error, stdout, stderr) => {
            if (error) {
                console.error(logTag, '[NODE-ERROR]', error.toString());
                throw error;
            }
            if (stdout) console.log(logTag, '[NODE-STDOUT]', stdout.toString());
            if (stderr) console.error(logTag, '[NODE-STDERR]', stderr.toString());
        });

path variable: D:\git\mongoclient_packaged\packaged\Mongoclient-win32-x64\resources\app.asar.unpacked\bin\node.exe

mainPath variable: D:\git\mongoclient_packaged\packaged\Mongoclient-win32-x64\resources\app.asar\app\main.js

Both are okay, but this execFile fails with:

[MONGOCLIENT] [NODE-ERROR] Error: Command failed: D:\git\mongoclient_packaged\packaged\Mongoclient-win32-x64\resources\app.asar.unpacked\bin\node.exe D:\git\mongoclient_packaged\packaged\Mongoclient-win32-x64\resources\app.asar\app\main.js
module.js:327
    throw err;
    ^

Error: Cannot find module 'D:\git\mongoclient_packaged\packaged\Mongoclient-win32-x64\resources\app.asar\app\main.js'
    at Function.Module._resolveFilename (module.js:325:15)
    at Function.Module._load (module.js:276:25)
    at Function.Module.runMain (module.js:441:10)
    at startup (node.js:139:18)
    at node.js:990:3

If I use fork instead, calling main.js works but then I get stuck where you've described (wrong path names etc...) But of course, we would not want to rely on user's node process and don't want to use fork.

I've shared this work as another branch called dev, and it would be really appreciated if you have a chance to take a look at it.

[HZSamir]

Hello again, so I looked at your branch and fiddled with it a bit, and here's what I got:

• Running MongoDB never seems to work for me, I usually just remove the beginStartingMongo and just call startNode. I have no idea why. It just stays stuck, with no error log.
• You have to fix the paths for the executables, otherwise they point to the wrong folder, like so:

// Mongo
let path = join(appRoot, '..', 'app.asar.unpacked',  'bin', 'mongod');

//Node 
let path = join(appRoot, '..', 'app.asar.unpacked', 'bin', 'node');

• Then I encounter a problem with the Fibers module. Basically I have to run node-gyp configure && node-gyp build and put the built binary inside a folder named win32-x64-53. Otherwise, the executable fails.

• And finally this:

So I wanted to suggest something: Instead of starting from scratch like you are, why don't we pick up this guy's work. He basically automated the whole process of bundling the app, copying the node_modules folder etc... It just seems like a logical step.

The repo hasn't been updated in ages. You can check however my fork of it. I just fixed some issues that popped up when Electron was updated. It works, creates an app for any platform, except without Asar support.

Let me know what you think of it. Good day.

[rsercano]

Hi @Unforgiven-wanda

You need to follow steps from here to package application (and of course you need to have a meteor application source code), differently from that guide, you need to run node packager.js in that dev branch. (Edit packager.js as you wish)

The repo you've mentioned has a lot of overhead to me, and I don't want some implementations from there, such as sockjs etc.. And it already has no Asar support ;(

I guess it's too complicated to put Asar since we can't resolve all dependencies that meteor is being used and can't change all paths there. I feel like getting stuck at this point

What do you think ?

Cheers

[HZSamir]

I'm just as stumped as you are. The thing is, I think I could be able to figure it out, if I invest enough time. Unfortunately I can only work on it whenever I have some free time at home (which is not much) and so I haven't been able to make any progress at all.

As for this issue in general, I was able to somewhat find a workaround to make it work on Windows without packaging to Asar: By going over the dependencies and

• removing all unused deps first
• cleaning all unused files in node_modules folders (with a nifty library called modclean)
• flattening the node_modules folder to reduce nested deps as much as possible.

Thanks to this I was able to significantly reduce the paths length, from somewhere around 280 to 130. Granted, there are still a lot of files, resulting in horrendous install times, but at least this way, you no longer have to worry about Windows.

If you're interested in the step-by-step process, let me know.

[rsercano]

I would like to hear more @Unforgiven-wanda, I totally agree with you and we're in the same boat. I really have too little time for this (I'm developing Mongoclient at my spare time at home usually), unfortunately have to work full-time in a company for my life :) I would like to have vice versa situation.

Apart from that, you can help me developing Mongoclient as well at your spare time, maybe it comes to a partnership in a further time if you're interested in.

As soon as you share the details of steps to shorten paths for windows I would like to add it to this repo. So maybe someone at somewhere can use this information :)

[HZSamir]

I'm copying exactly what I wrote on Gitter, explaining this to others:

Yes, I have run into this same issue. One way to deal with this issue is to package your app into an ASAR file, however this will not work with Electrify. The trouble comes from the spawn method. Electrify relies on it to instantiate both Meteor and Mongo, however the function is incompatible with ASAR (see their docs for more info). The way I found to deal with this issue is to flatten all my deps inside the packaged application, and after much trial and error I have come across the following three packages : flatten-packages, flatten-deps and modclean. Mind you, it is a somewhat dirty method, but it gets the job done, as the node_modules deps are the ones responsible for the long paths. My method is as follows: First you must install all three packages globally. Then go to your packaged app folder, say my/app/path/resources/app and run sudo modclean -n default:safe,default:caution. This will delete all folders and files that are unnecessary for productions, such as empty folders, test files and the like, again, check the docs for more info Then go to every folder as follows and run these commands: my/app/path/resources ==> run flatten-packages Same thing for the following folders: my/app/path\resources\app\app\programs\server, my/app/path/resources/app/app/programs/server/npm (mind the slashes, I've messed up some of them)

Then go to my\app\path\resources\app\app\programs\server\npm\node_modules\meteor. It should contain all packages you've installed from Meteor. You will enter each folder in order, and call flatten-deps (careful, call flatten-deps here and not flatten-packages as the latter kinda messes up everything). Do this for every packages in this folder Once you're done, go up to my\app\path\resources\app\app\programs\server\npm\node_modules, call modclean again with modclean -n default:safe,default:caution then flatten-packages If you've done everything exactly as I've explained, the paths should be considerably shorter. In my case they went down from 280 in length to somewhere around 130.

Now You can follow the same steps for the bundled project you output. It should work. I had a script at some point for this that I lost, sorry :/

[rsercano]

Thanks for the explanation @Unforgiven-wanda I'll do same to Mongoclient as well