node-gdal
node-gdal copied to clipboard
Avoid storing GDAL sources in git
I think we should move the gdal sources out of git versioning to avoid the noise and git bloat when the bundled gdal version is upgraded (e.g. the diff for https://github.com/naturalatlas/node-gdal/compare/0.3.x is too big to preview).
Options include:
- storing only the compressed tarball (this is what we do for node-sqlite3)
- dynamically fetching the tarball
Given that gdal-1.11.1.tar.gz
is still > 10 MB I think dynamically fetching would be best.
To support all the different platforms/compilers we do, we need to patch GDAL. I'm not aware of any great approach to patch
in a cross-platform way. I suppose we could keep a copy of pre-patched files that we replace after the GDAL source is extracted, but I'm not sure that is ideal. This brings about a bigger conversation about dependencies and building. Here's something I propose (reasoning at bottom):
Near term:
- New repo, like we've been kicking around:
libgdal-gyp
- Publish it to NPM, so it all gets tar.gz'd and we keep the diff madness out of node-gdal. This approach would also reduce the download size, because we don't need download tests, readmes, etc that are never used (currently stripped by libgdal.sh)
- Use node-pre-gyp to pre-build the library. This will reduce build times of node-gdal.
- Version it to match the version of GDAL (e.g.
1.11.1-0
). Dependents like node-gdal can then require the default version of GDAL they want.
Long term:
- Move to a less-monolithic setup.
- Split out the other deps with this style:
libexpat-gyp
,libproj-gyp
,libgeos-gyp
. This would further-reduce build times and allow for node bindings to these particular libs, withoutnode-gdal
even being in the picture.
Other thoughts:
- This approach would allow us to support a
gdal-shared
NPM package that doesn't depend on bundled code/libraries at all. For people who know they always want to use system-wide GDAL, this would be largely beneficial. - We're getting into a dependency and build nightmare.
- I still have no idea what goes on for Windows building (and where that takes place).
- Building node-gdal w/GDAL bundled w/mapnik doesn't seem like a great idea. I get how it makes sense for Mapbox, but not for others. It locks us into whatever mapnik is using, and ties gdal development to mapnik development. Right?
- Publish a compatibility table for people using
node-mapnik
andnode-gdal
in the same project.
This would be a big undertaking, but I feel we're digging ourselves into a hole when we keep compounding approaches that add unrelated repos and dependencies on people + manual processes to the mix.