datproject-discussions
datproject-discussions copied to clipboard
dat-ecosystem-archive
Dat Module Wishlist
This issue is to discuss ideas for dat modules that would be awesome to have. If you want to write a module leave a comment to notify others that you are working on it (preferably also leave a link to the repo).
I'll keep this post updated to link to all of the modules (until we have too many to fit here).
To learn more about dat modules check out the Importing data guide
If a module doesn't have a link w/ it it means nobody is working on it yet
Read about Import Methods to determine which type of module should be used for each case.
List
(not in any particular order)
In Progress/Done
- Google Drive - started
- Google Cloud Storage - started
- Postgres - done (experimental)
- Heroku Postgres - done (experimental)
- PouchDB - initial working version
- ElasticSearch - proof of concept
- MongoDB - mongodown (alpha), node-mongs (alpha)
Up for grabs
- CouchDB
- CartoDB
- MySql
- MSSQLServer
- Redis
- Hadoop
- MongoDB
- Google BigQuery
- Amazon RedShift
- Amazon SimpleDB
- Amazon DynamoDB
- ESRI ArcGIS/ArcServer
- Microsoft Access
- Microsoft Azure Table Storage (maybe use this?)
- Microsoft Azure Blobs (details)
- IBM SQL Database for Bluemix
- Socrata Data Portals
- Elasticsearch
- FTP
- HTTP (open directory spider?)
- Filesystem (maybe w/ filewatcher support to update hashes?)
- Dropbox
- S3
- Spideroak
- Box
- IPFS
- Google Fusion Tables
- OpenStack Swift
- Ceph
- HDFS
- HDF5 (also see hdf5.node and h5py)
- Google Cloud SQL
- RethinkDB
- CKAN API, in particular through datahub
- firebase
- Neo4j
- NodeRED
Some other storage possibilities (in no particular order, and I am not working on any of these):
- Spideroak
- Box
- IPFS
- Google Fusion Tables
- OpenStack Swift
- Ceph
- HDFS
- HDF5 (also see hdf5.node and h5py)
- Google Cloud SQL
@webmaven awesome list, thanks! I definitely have @jbenet on the hook to write an IPFS backend once it's ready
Here's one uploader for ElasticSearch based on the module by @missinglink: https://github.com/astro/dat-elasticsearch-upload
Support the CKAN API, in particular through http://datahub.io (part of CKAN - Open Knowledge Foundation) - Latest docs here: http://docs.ckan.org/en/latest/api/index.html
+1 on CKAN datastore API support, or even a CKAN extension that uses Dat instead of CKAN's datastore.
Not sure if this fits here, but I just created vinyl-dat to read/write dat records in a gulp workflow. It's basically just a way to convert dat records into vinyl files. I think it still needs some work and possibly a better way to just let dat do conversions on contents (like choosing csv, json, etc...)
I think this should work for redis as a backend https://www.npmjs.org/package/redisdown
I am going to have a look at CKAN as an import/export target as well as a possible storage back-end. ckanDOWN: https://github.com/finnp/ckanDOWN
Would https://www.npmjs.org/package/gdal be a usefull lib for dat to connect to a huge number of formats (http://gdal.org/ogr_formats.html)?
@pvgenuchten that is awesome, thanks for linking it. I'm not sure what the best place to plug gdal into dat would be at the moment but I'll think about it. It's also great that the gdal module is using node-pre-gyp to make it easy to install cross-platform
We'd be interested in throwing http://figshare.com/ in the ring here, also have some devs who could help out :)
So this is a pouchdb want originally but applies to dat.
Leveldown can be tricky to get working on all platforms and while pre-built ones ones are helpful there are always going to be weirder platform combinations.
It would be great to have a pure js leveldown backend (for node) that was not restricted to memory.
Being able to install dat or pouchdb with no c deps would be super useful.
Medea seems the best of what is out there but has a requirement that keys be kept in memory which combined with the node.js hard 2GB mem limit means that there are hard limits on the datasets you'd be able to put into it.
So really the thing that would be super helpful is a pure js nodejs db with spill to disk.
Check out http://geogig.org/
"GeoGig is an open source tool that draws inspiration from Git, but adapts its core concepts to handle distributed versioning of geospatial data."
Has many of the basic one not been covered? :) As seen here https://github.com/rvagg/node-levelup/wiki/Modules#storage
Or am I am wrong? I am a little confused because how is this dat related, isn't is more of Leveldb related, that is needs more adapters?
Or is there a good example on a plugin there is written?
@calvinmetcalf If you're looking to build a pure JS levelDB this might be a good place to start -- https://github.com/rvagg/leveljs
Ooo I like, thanks On Oct 23, 2014 2:04 PM, "Yoshua Wuyts" [email protected] wrote:
@calvinmetcalf https://github.com/calvinmetcalf If you're looking to build a pure JS levelDB this might be a good place to start -- https://github.com/rvagg/leveljs
— Reply to this email directly or view it on GitHub https://github.com/datproject/discussions/issues/5#issuecomment-60281261 .
any interest to RDF / Linked Data support? I can work on a dat-marmotta implementation for testing it.
@wikier RDF / Marmotta here, please! We'll need it for http://transformap.co
so @almereyda, should we create an issue to organize the work, define the goals, etc?
@wikier Yes, of course! We'll be squatting the CouchDB Meetup in Berlin next week, where I'm also inviting @finnp right now, to precise the requirements of the TransforMap data integration pipeline.
Please feel free to get started with your perspective, but I will only then be able to respond meaningfully.