naturalatlas
Standard PostgreSQL support?
I have noticed that the standard PostgreSQL OGR format is not standard and it has now become a requirement for the tool I am trying to build.
I wasn't sure how to do this so attempted to manually edit this https://github.com/naturalatlas/node-gdal/blob/master/deps/libgdal/libgdal_formats.gypi with an entry for 'pg'. I then created a new ogr_pg.gyp file in the gyp-formats folder, and in it i reference the files in https://github.com/naturalatlas/node-gdal/tree/master/deps/libgdal/gdal/ogr/ogrsf_frmts/pg
I then tried
npm install node-gdal --build-from-source
And got a horrible number of errors and realised that I should use:
npm install node-gdal --build-from-source --shared_gdal
But again received errors during the build.
I have now just read all the closed and open issues about 'builds' and found this comment: https://github.com/naturalatlas/node-gdal/issues/88#issuecomment-74834030
Which makes me think the routine is to build node-gdal using
npm install node-gdal --build-from-source
And then have the deplibs folder containing the folder structure as outlined in that comment and then running
npm install --build-from-source --msvs_version=2013 --shared_gdal=c:/projects/node-gdal/deplibs/
but I do not know where the contents for deplibs comes from. It doesn't seem to be created during a standard build routine.
Really sorry for asking such a NOOB question. Am sure it is super simple and am just overlooking a solution on Windows.
BTW using Windows 7 x64 and have MSVS2013
I took a look at this but it's not that simple.
Since node-gdal aims to be self contained libpq would have to be pulled in to make this work.
Maybe @brandonreavis or @brianreavis could add their $0.02 if they want to include support for PG.
I took a first stab at https://github.com/naturalatlas/node-gdal/compare/issue-132?diff=split&name=issue-132
It's a start, but
- everything hardcoded to my local paths
- build is still failing, because of some missing references
Feel free to delete the branch, if you decide against PG support (@brandonreavis @brianreavis).
@TimJMartin Will you always have postgres on the machine you're trying to use node-gdal on, or are you looking for a self-contained build that can work anywhere? For the latter, like @BergWerkGIS said, we're going to have to pull in libpq and all its dependencies and build them from source to static link (as far as I know). Refs https://github.com/naturalatlas/node-gdal/issues/75 (which I'd love to have, but I'm not sure is practical right now)
For the first option, we can at least make npm install node-gdal --build-from-source include postgres support if it finds it one the system. What errors are you getting when running that? Really unfamiliar with Windows building... definitely @BergWerkGIS's turf.
@brianreavis @BergWerkGIS
Yes people would have to have PostgreSQL and PostGIS installed. So hopefully that makes life a lot easier to include it as a format option.
I am aiming to use the node module 'pg' to setup the database tables then load in the data using
--config PG_USE_COPY YES
As our data is pretty big. The reason I couldn't just use PGDump is because I would then like to enable export back out of PostGIS.
It is more than likely it is my routine that is wrong.
I have a package.json with the dependencies pointing at
"dependencies": { "gdal": "naturalatlas/node-gdal#win-node-4.2.1" }
Then I do
npm install gdal --build-from-source --msvs_version=2013
This builds successfully, i then edit the
libgdal_formats.gypi deps/libgdal/gyp-formats/ogr_pg.gyp
And then run
npm install gdal --build-from-source --msvs_version=2013
But this seems to overwrite my edits to the files above by downloading all the source files again.
So as I said it is probably my routine that is the issue.
@TimJMartin
The branch you would have to work against currently is issue-132:
"dependencies": {
"gdal": "naturalatlas/node-gdal#issue-132"
}
I made it compile with the library and headers that come with a default Postgres install.
how I did it:
Paths in the gyp files are still hardcoded to my Postgres install: C:/Program Files/PostgreSQL/9.4/lib.
I used VS2015 and I'm not sure about the consequences when Postgres libs are built with another compiler. Might lead to unpredictable behavior.
SET PATH=C:\Python27;%PATH%
SET PATH=set path=C:\Program Files\PostgreSQL\9.4\bin;%PATH% <- for pg_config
git clone https://github.com/naturalatlas/node-gdal.git
cd node-gdal
git checkout issue-132
npm install --build-from-source --msvs_version=2015
Directly within the clone directory I used this:
var gdal = require('./');
//console.log(gdal.drivers.getNames());
try{
var ds = gdal.open("PG:dbname='XXXX' host='localhost' port='5432' user='XXXX' password='XXXX'");
ds.layers.forEach(function (lyr){
console.log(lyr.name);
});
}
catch(err){
console.log(err);
console.log(gdal.lastError);
}
(just referencing this here... forgot it existed: https://github.com/naturalatlas/node-gdal/issues/20)
@BergWerkGIS
Just tried your routine and it seemed to have worked for the new pg but got the following build error:
libgdal_ogr_wasp_frmt.vcxproj -> C:\Temp\issue132\node-gdal\build\Release\\libgdal_ogr_wasp_frmt.lib
gyp ERR! build error
gyp ERR! stack Error: `C:\Program Files (x86)\MSBuild\12.0\bin\msbuild.exe` failed with exit code: 1
gyp ERR! stack at ChildProcess.onExit (C:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\lib\build.js:2
70:23)
gyp ERR! stack at emitTwo (events.js:87:13)
gyp ERR! stack at ChildProcess.emit (events.js:172:7)
gyp ERR! stack at Process.ChildProcess._handle.onexit (internal/child_process.js:200:12)
gyp ERR! System Windows_NT 6.1.7601
gyp ERR! command "C:\\Program Files\\nodejs\\node.exe" "C:\\Program Files\\nodejs\\node_modules\\npm\\node_modules\\node
-gyp\\bin\\node-gyp.js" "build" "--fallback-to-build" "--module=C:\\Temp\\issue132\\node-gdal\\lib\\binding\\gdal.node"
"--module_name=gdal" "--module_path=C:\\Temp\\issue132\\node-gdal\\lib\\binding"
gyp ERR! cwd C:\Temp\issue132\node-gdal
gyp ERR! node -v v4.2.1
gyp ERR! node-gyp -v v3.0.3
gyp ERR! not ok
node-pre-gyp ERR! build error
node-pre-gyp ERR! stack Error: Failed to execute 'node-gyp.cmd build --fallback-to-build --module=C:\Temp\issue132\node-
gdal\lib\binding\gdal.node --module_name=gdal --module_path=C:\Temp\issue132\node-gdal\lib\binding' (1)
node-pre-gyp ERR! stack at ChildProcess.<anonymous> (C:\Temp\issue132\node-gdal\node_modules\node-pre-gyp\lib\util\c
ompile.js:83:29)
node-pre-gyp ERR! stack at emitTwo (events.js:87:13)
node-pre-gyp ERR! stack at ChildProcess.emit (events.js:172:7)
node-pre-gyp ERR! stack at maybeClose (internal/child_process.js:818:16)
node-pre-gyp ERR! stack at Process.ChildProcess._handle.onexit (internal/child_process.js:211:5)
node-pre-gyp ERR! System Windows_NT 6.1.7601
node-pre-gyp ERR! command "C:\\Program Files\\nodejs\\node.exe" "C:\\Temp\\issue132\\node-gdal\\node_modules\\node-pre-g
yp\\bin\\node-pre-gyp" "install" "--fallback-to-build"
node-pre-gyp ERR! cwd C:\Temp\issue132\node-gdal
node-pre-gyp ERR! node -v v4.2.1
node-pre-gyp ERR! node-pre-gyp -v v0.6.12
node-pre-gyp ERR! not ok
Failed to execute 'node-gyp.cmd build --fallback-to-build --module=C:\Temp\issue132\node-gdal\lib\binding\gdal.node --mo
dule_name=gdal --module_path=C:\Temp\issue132\node-gdal\lib\binding' (1)
npm ERR! Windows_NT 6.1.7601
npm ERR! argv "C:\\Program Files\\nodejs\\node.exe" "C:\\Program Files\\nodejs\\node_modules\\npm\\bin\\npm-cli.js" "ins
tall" "--build-from-source" "--msvs_version=2013"
npm ERR! node v4.2.1
npm ERR! npm v2.14.7
npm ERR! code ELIFECYCLE
npm ERR! [email protected] install: `node-pre-gyp install --fallback-to-build`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] install script 'node-pre-gyp install --fallback-to-build'.
npm ERR! This is most likely a problem with the gdal package,
npm ERR! not with npm itself.
Does this look like a problem with using MSVC2013 rather than 2015?
I will try and launch an AWS EC2 windows box and try it on a clean machine of PostgreSQL 9.4 and MSVC2015
I see you have npm ERR! node v4.2.1.
I tried it with 0.10.40 and 0.12.7.
I branched off master and not the win4 branch.
@BergWerkGIS ok just fired up an EC2 so will get 0.12.7 installed and MSVS 2015 Community Edition
@BergWerkGIS so got it working using your routine
I used MSVS 2013 Express as described in the node gyp requirements for Windows 7.
Installed PostgreSQL 9.4 64bit and PostGIS 2.2 64bit
Installed Node 0.12.7
Followed your instructions and the build successfully worked as did your script to connect and see layers in the database.
That is excellent!
So would the next steps be to include that along with the 4.2.1 branch and do a release at some point?
So would the next steps be to include that along with the 4.2.1 branch and do a release at some point?
Yes, suppose so, but I think it might take a while, as it would require some additional effort. @brianreavis I'm all ears on your opinion about this.
I'm tight on available free resources right now, so might only be able to help out again for small Windows specific tasks.
- disable Postgres support by default. How big is the demand? Looking through the issues this seems to be the first request for Postgres support?
- make it available when building from source, e.g.
npm install --build-from-source --pg_enable=1? - incorporate this change in the gyp files
- check if
pg_configis available. If not continue without PG support or bail out? - parse the output of
pg_config - create new tests for Postgres
- setup up test database during test
- change CI scripts (travis, AppVeyor) to setup test db and run Postgres tests
- anything else?
@BergWerkGIS sorry for slow reply
totally understand. You guys have been super helpful and I want to say a big thank you for all your hard work to get vsigzip sorted and the electron version done.
If and when you get time to look at this let me know as I can test a version