tuttle
tuttle copied to clipboard
lexman
Make for data
- [ ] `tuttle invalidate -q` => queries if there is something to do. Should behave like make -queries - [ ] `tuttle clean around` => should remove everything in...
For exemple, a workflow retreive images from an SD card, or anonymizing data. This goaled should be archieved with the current model if working with wildcards or dynamic dependencies Needs...
- [x] ftp - [ ] sftp - [x] https - [ ] ssh file - [x] hdfs - [x] S3
A standard way to declare constants, for exemple yaml or .ini Could be used for Pre-run dependancy graph Processes using constants should declare them as dependancies, in order to invalidate...
Javascript is a widely used language for dealing with data, espacialy for making visalizations
tuttle uses dh-virutalenv to create .deb packages with extact control over python dependencies. If this packaging system is ok for the debian community, we could submit some packages to the...
PROV model ( http://www.w3.org/TR/prov-overview/ ) is a normalized way to describe where data come from, how it has been created, and how it has evolved, etc. tuttle could export where...
- [x] implementation of postgres resources - [x] implement other resources than tables : views, sotred procedures... What about indexes ? - [ ] implement sub bloc from tables with...
- [ ] finalise interfaces for both resources and processors - [x] detailed documentation for the core resources and processors - [ ] a good documentation of how to write...
An auto processor should be able to understand what to do according to the input and the ouput resources For exemple : insert a csv into a table, extract fils...