openvenues
Build libpostal as part of the installation process
Is it possible to build libpostal as part of the npm install process? It will save the effort of installing libpostal separately.
Usually when dynamically linking to a native library, it's either assumed that the library is installed separately, or that the dependency is included with the binding. Let's call these the "lean repo" and the "fat repo" approaches respectively. node-postal is an example of a lean repo, whereas a fat repo would be something like node-snappy.
libpostal is a bit trickier than a library like Snappy because it's not just software - there are also data/model files which need to be downloaded from the web. Because said model files take up a few GB of disk space, we allow the --datadir option to configure so the user can e.g. select a directory on a volume that has sufficient storage. If no config option is specified, it defaults to something like /usr/local/share on many systems.
At a minimum, any one-liner version of npm install that also installs libpostal would need to allow specifying the datadir, and should probably check for an existing libpostal installation. Some users may for instance already have the Postgres extension or another binding installed and configured with a given datadir, and would expect that npm install does not then reinstall libpostal, download the model files again, and point the datadir to /usr/local/share, affecting the other binding as well.
IMHO it's not worth implementing something complicated to save 5 lines of (very standard) shell, but will accept pull requests as long as the new behavior is optional and non-default.
Thanks for the response @thatdatabaseguy . I do wish there's a simpler one-liner solution, but it seems this cannot be achieved without complicating the installation logic.
Maybe we could provide 3 installation strategies:
- Prebuilt binaries (with node-pre-gyp)
npm install node-postal - Build against libpostal static sources
npm install node-postal --build-from-source(fallback for first strategy) - Build against shared lib
npm install node-postal --build-from-source --shared_libpostal(actual strategy)
In the futur nan could be replaced by the upcoming and experimental N-API:
https://medium.com/the-node-js-collection/n-api-next-generation-node-js-apis-for-native-modules-169af5235b06
I can help if you are interested!
This sounds like the perfect balance.
From: Jérôme Desboeufs [email protected] Sent: Wednesday, August 16, 2017 4:16:32 AM To: openvenues/node-postal Cc: Mengxuan Xia; Author Subject: Re: [openvenues/node-postal] Build libpostal as part of the installation process (#10)
Maybe we could provide 3 installation strategies:
- Prebuilt binaries (with node-pre-gyp) npm install node-postal
- Build against libpostal static sources npm install node-postal --build-from-source (fallback for first strategy)
- Build against shared lib npm install node-postal --build-from-source --shared_libpostal (actual strategy)
In the futur nan could be replaced by the upcoming and experimental N-API: https://medium.com/the-node-js-collection/n-api-next-generation-node-js-apis-for-native-modules-169af5235b06
I can help if you are interested!
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/openvenues/node-postal/issues/10#issuecomment-322699430, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AAnTR4oCN89rOVxKxU18cgliOZ9MR7QXks5sYqVggaJpZM4LRDzg.
That sounds good. Not quite sure if static linking would play well with our data downloads though. Part of the make process in libpostal is downloading ~700MB of files that decompress into ~1.8GB on disk, which is more than most users are expecting out of something that they npm installed, and may necessitate placing the libpostal data files in a non-default location (like an EBS volume). If this were part of Linux the data download would be an "are you sure [y/N]?" moment.
Currently LIBPOSTAL_DATA_DIR is defined at compile-time so that default value would be baked in to the static libpostal.a. The thought there was to design a process that would work well with the standard configure/make/make install workflow familiar to most *nix users.
I'd be fine with downloading/compiling libpostal as part of npm install and just recommending that users pass through a --datadir option.
As far as node-pre-gyp, I looked at it before and it seems like a no-brainer for C/C++ extensions where all dependencies were contained within the project, but not ideal for our case as pre-built binaries would entail either forcing a datadir location on the user or doing some rearchitecting of libpostal/node-postal to allow datadir to be specified at run-time.
Happy to accept pull requests.
@jdesboeufs I think the main backer of LibPostal, Mapzen, has announced closure :(. Therefore I expect few are working on this supplemented by Mapzen. So your offer to help (almost a year ago) is perhaps more necessary than ever. There are also LibPostal buildpacks for Heroku/Herokuish environments. Improving or updating them is probably a similar type of work: https://github.com/homelight/heroku-buildback-libpostal (see forks).
If you've found another libpostal alternative in the meantime, please share!
Hey folks. I found another heroku-libpostal-buildpack on gitlab that has a prebuilt tarball of the install. I haven't used it yet but thought I would share anyway. I'll send an update after I get something deploying.
https://gitlab.com/NoahGray/libpostal-buildpack-prebuilt
