geopipes

[joehand]

From @missinglink on July 21, 2014 21:29

Hey @maxogden, team.

As we discussed the other day at nodeconf in London, I'm currently leading a project at mapzen to create a set of curated stream libraries for working specifically with geographic datasets.

My personal interest in creating the libraries is to facilitate importing of large amounts of POIs in to elasticsearch for our opensource/opendata geosearch engine called Pelias.

However, outside that; it'd be great to have a set of stream libraries for dealing with these large geo datasets which can empower other developers to build their own geo pipelines.

I've just created the geopipes organisation on github today and I added an elasticsearch backend that should be compatible with dat, if you're interested in that sort of thing. https://github.com/geopipes/elasticsearch-backend

disclaimer: I only wrote that this morning, it has pretty full test coverage but I haven't given it a thorough beating yet.

Over the next couple of weeks I will add the repos for openstreetmap, geonames and quattroshapes as well as streaming parsers for osm pbf and shapefiles. (mostly written by others)

max asked me to open a ticket to discuss if we can find some way of working together and to make you aware of what we're working on.

Copied from original issue: maxogden/dat#140

[joehand]

From @missinglink on July 21, 2014 21:34

This is the issue I mentioned about substacks osm-pbf-parser library which is preventing me from adopting it. https://github.com/substack/osm-pbf-parser/issues/5

at the moment, I am using this instead: https://github.com/marook/osm-read/issues/22

[joehand]

From @jden on July 21, 2014 22:3

@missinglink sounds super cool! Is there a good primer or anything written which explains the geopipes idea in more detail?

[joehand]

From @missinglink on July 23, 2014 16:58

Hey @jden, sorry I haven't got a primer written yet. BUT!, I just shipped a module for working with geonames: https://github.com/geopipes/geonames-stream

Is this sort of stuff compatible with dat?

[joehand]

From @missinglink on July 23, 2014 18:42

osm: https://github.com/geopipes/openstreetmap-stream

[joehand]

From @jden on July 23, 2014 18:45

Okay - I think I understand more now. Geopipes is more of a set of tools, rather than a format or a protocol, is this correct?

[joehand]

From @missinglink on July 23, 2014 18:47

yep. it's simply a project to create a set of streaming libraries for geo related datasets

[joehand]

From @maxogden on July 24, 2014 13:13

I just wrote up an initial draft of a guide on importing data into dat https://github.com/maxogden/dat/blob/master/docs/importing.md

I think geopipes is definitely gonna be useful

[joehand]

From @maxogden on July 24, 2014 13:13

Also: https://github.com/datproject/meta/issues/5

[joehand]

From @ryanramage on December 2, 2014 16:52

Wrote this recently, https://www.npmjs.org/package/reproject-stream noting here as there is no 'official list' of these types of tools. Maybe someone will find useful.