Addon-WG: Bundle CPython into Addon Tooling

[eljefedelrodeodeljefe]

TL;DR

Bundling Python on unix is really simple . Exchanging data (e.g. objects) as well. Feels a bit like compiling the whole Python-lang though and runs for 2+ mins. Was attempting this at eljefedelrodeodeljefe/node-cpython in order to run py-libs natively.

Disclaimer It's a working API but not tested currently (left that for vacation next week)

I am doing lot's of native addon building and figured out that compilation can be a real pain. I was investigating on this for quite a while and wanted to propose shipping Python.h and exposing a high level API (streams as well) to JS in order to be able to abstract gyp. It would render you to not just call .exec() on it, but having it in cpp-land.

@bnoordhuis would that be something worth a discussion? (pointing at you since you were talking about this in the last CTC and I liked your opinions in Berlins Code and Learn)

I think it has been mentioned a lot that gyp will not change and especially the python-2.7 dependence is quite annoying. With the above approach, one could just choose which version to run from JS. Also it would enable one to slowly deprecate some of the gyp or python tooling, by keeping the API, but exchanging the below implementations with JS stuff.

Maybe this would ease the multitude of node-gyp problems on Windows, as an example.

[bnoordhuis]

Interesting! I spent some time this week on a similar idea where I bake gyp into a minimal python executable for distribution. My thinking was that it:

1. Would get rid of the issue where people have a conflicting copy of gyp on their PYTHONPATH, and
2. Make things a little easier for Windows users because it would just be part of the installer.

I'm not married to the concept though. Can you describe how integration would work with your approach?

One critical property that needs to keep working is the ability for gyp files to run python code and scripts because that's used quite a bit for feature sniffing and platform detection.

For example, here is the arch detection script that V8 invokes from here . A few lines below it shells out to python. While V8 is something of a special case, the use of arbitrary python code in gyp files is not, for better or worse.

[whitlockjc]

Make sure when you look at @bnoordhuis' sample that you read my comment. We finally got gyp embedded properly but there are some cases we cannot account for without some extra effort. I'd love to help out more.

[bnoordhuis]

I'm at a point now where I can generate a Makefile for libuv on both OS X and Linux. Progress!

[eljefedelrodeodeljefe]

@bnoordhuis I am still on this (sorry for the late response). Made some progress there as well, w/ the lib being usable, I guess.

TL;DR

Does it make sense to get rid of toolchains in favor of a generic build.js though abstracting compiler toolchains in js.

(cont.) However, I was reading up on Windows Kernel stuff and some of @piscisaureus tickets in the core repo concerning windows build issues etc. maybe all of them somehow related to this gyp/make/VS toolchain problem. Due to being new to Windows development, it was really hard for me to figure out how to properly compile the core repo, let alone node-gyp related stuff.

So, I don't want to make this a major rant of "all the existing build tools suck", really I don't, but I was thinking of abstracting toolchains w/ node directly. Basically write a lib platform_tools that does handling compilers for you. I came to the idea after the realisation, that I will never get friends w/ batch and VS toolchains and looked into working w/ cl.exe directly.

So this might be a very long haul, but wouldn't this render autotools and eventually maybe also every other toolchain obsolete? I was starting this here or see this gist

This might be nothing for core, but maybe worth a try for some external lib. How was your progress concerning the python stuff? Did it land somehere?

Happy new year, after your midnight.

Edit:

example usage:

const tools = require('platform_tools')
// options:
// below are the defaults
let options = {
     showVersion: false, // shows the compiler version, when starting. win32 this will be shown in any case
     verbose: false,
     includes: [
            // paths to includes
     ], // default: none; win: concat of /I, unix: concat of -I
  suppressWarnings: false, // win: /w, unix: -w
  suppressAll: false,
  showAllWarnings: false, // win: /Wall, unix: -Wall
  optimizations: {
        forSmallCode: false, // win: /O1
        forFastCode: false, // win: /O2
        disableOptimization: false, // win: /Od
        favorsSmallCode: false, // win: /Os
        favorsFastCode: false, // win: /Ot
        maxOptimization: false // win: /Ox
  }
}
// further description of the compiler flags see:
// win: [MSDN reference](https://msdn.microsoft.com/de-de/library/19z1t1wy.aspx), clang: [clang.llvm.org](http://clang.llvm.org/docs/UsersManual.html#command-line-options)

tools.compile('clang', 'plattform_tools_test_c.c', 'b.out', options, function () {
  console.log('did compile')
})