node-gdal icon indicating copy to clipboard operation
node-gdal copied to clipboard

Avoid storing GDAL sources in git

Open springmeyer opened this issue 10 years ago • 1 comments

I think we should move the gdal sources out of git versioning to avoid the noise and git bloat when the bundled gdal version is upgraded (e.g. the diff for https://github.com/naturalatlas/node-gdal/compare/0.3.x is too big to preview).

Options include:

  • storing only the compressed tarball (this is what we do for node-sqlite3)
  • dynamically fetching the tarball

Given that gdal-1.11.1.tar.gz is still > 10 MB I think dynamically fetching would be best.

springmeyer avatar Nov 07 '14 15:11 springmeyer

To support all the different platforms/compilers we do, we need to patch GDAL. I'm not aware of any great approach to patch in a cross-platform way. I suppose we could keep a copy of pre-patched files that we replace after the GDAL source is extracted, but I'm not sure that is ideal. This brings about a bigger conversation about dependencies and building. Here's something I propose (reasoning at bottom):

Near term:

  • New repo, like we've been kicking around: libgdal-gyp
  • Publish it to NPM, so it all gets tar.gz'd and we keep the diff madness out of node-gdal. This approach would also reduce the download size, because we don't need download tests, readmes, etc that are never used (currently stripped by libgdal.sh)
  • Use node-pre-gyp to pre-build the library. This will reduce build times of node-gdal.
  • Version it to match the version of GDAL (e.g. 1.11.1-0). Dependents like node-gdal can then require the default version of GDAL they want.

Long term:

  • Move to a less-monolithic setup.
  • Split out the other deps with this style: libexpat-gyp, libproj-gyp, libgeos-gyp. This would further-reduce build times and allow for node bindings to these particular libs, without node-gdal even being in the picture.

Other thoughts:

  • This approach would allow us to support a gdal-shared NPM package that doesn't depend on bundled code/libraries at all. For people who know they always want to use system-wide GDAL, this would be largely beneficial.
  • We're getting into a dependency and build nightmare.
    • I still have no idea what goes on for Windows building (and where that takes place).
    • Building node-gdal w/GDAL bundled w/mapnik doesn't seem like a great idea. I get how it makes sense for Mapbox, but not for others. It locks us into whatever mapnik is using, and ties gdal development to mapnik development. Right?
  • Publish a compatibility table for people using node-mapnik and node-gdal in the same project.

This would be a big undertaking, but I feel we're digging ourselves into a hole when we keep compounding approaches that add unrelated repos and dependencies on people + manual processes to the mix.

brianreavis avatar Nov 07 '14 21:11 brianreavis