CartoDB
Import API Streaming [feature Request]
Why is the import api limited to a fileName and or URL. Why can we not just pass a stream of raw data as an option? I would love this to exist so I can get rid of my own implementation.
So I was able to implement about 90% of this. But it looks like the main problem is that formData in the request library require having a knownLength to encapsulate the formData in a the lastBoundary. This is a real pain in the but as you need to process the stream once for length and then again to flow through this library.
Why can't the server accept streamed data of an unknown length? At one point I had this working via this code . Anyway that too appears to no longer work. (Tried to import most of that code to here to make Import.prototype.stream) .
Any ideas are welcomed as saving the file to /tmp/ to then use Import.prototype.file feels so hackish and dirty.
For reference of other people with the same problems see:
- https://github.com/request/request/issues/316
- https://github.com/form-data/form-data/blob/master/lib/form_data.js#L71-L106
- https://github.com/form-data/form-data/blob/master/lib/form_data.js#L315-L336