hootenanny
hootenanny copied to clipboard
ngageoint
Add service to import data from remote URL vs uploading local file
This PR adds a /remote service endpoint to import data from a URL. It makes use of the /vsicurl/, /vsis3/, and /vsizip/ features of OGR. The /vsizip/ is used to simplify expectations and also to use OGR to read osm data (vs Hoot's custom OSM reader that doesn't currently support remote URLs to osm files. However, this adds some caveats to the supported osm that can be read with OGR. For example, by default OGR doesn't like the negative IDs that would be on features not yet in the OSM db returning this error Unsupported node id value (-1560244). Use OSM_USE_CUSTOM_INDEXING=NO.
The S3 url will use AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY for the username and password values. This link can be used to test s3://hootenanny-remote-upload-test/dcpoi_clip.zip, I can provide the creds in a DM.
The http/ftp are currently using Basic Auth creds embedded in the hostname (e.g. http://user:pw@hostname:port/) but this is not ideal and not sure if it's supported with https and sftp/ftps.
To test the ftp w/basic auth endpoint I used this command to serve the /hoot/test-files/ directory with the zip file.
python3 -m pyftpdlib -i 192.168.56.1 -u hoot -P hoottest
The UI changes add a "Remote URL" import type:
This choice then shows optional authentication fields:
