Streams for Kitura?

[crspybits]

My use case: I want to have my HTTP server be able to receive requests containing files, but not store those files on my server in any fashion. Instead, I want to stream those files off to other services. E.g., to Google Drive. My rationale for not storing files on my server has several motivations:

1. It ought to be more efficient to stream the incoming (uploaded) file off to something like Google Drive, without first storing that upload to a file in the server-local file system;
2. For purposes of server scaling, storing files in a local file system is not necessarily a good idea-- those files may be removed upon a server reboot and make recovery from errors more difficult;
3. I don't want to have to pay for temporary file storage on a server for these files.

What I'm looking for is a Swift implementation of Streams in the sense of Node.js. E.g., the standard HTTP request objects in Node.js (see https://nodejs.org/api/http.html#http_class_http_incomingmessage) implement a stream.Readable interface (see https://nodejs.org/api/stream.html#stream_class_stream_readable) which for example make a pipe method available where you can pipe the HTTP request to a writable stream. Busboy provides an example of this usage-- see http://schempy.com/2015/03/11/streaming_file_uploads_with_nodejs/

It looks like that at present Kitura doesn't support this kind of functionality. Is there interest in adding this? I'd be looking for architecture ideas, to do the development myself, and looking for code reviews on PRs. Any and all input would be accepted.

I've done some reading through the Kitura-net code and it looks like a place to start might be with the ServerRequest protocol, analogous to the way in which Node.js does this. Perhaps ServerRequest could optionally support streaming methods and properties. Or perhaps this could just be a different protocol, say, ReadableStream, which is implemented by some classes implementing ServerRequest. I'm not sure if a different server type would be necessary. e.g., comparable to the HTTP or FastCGI servers.

Please let me know your thoughts on this. This is not for the company I work for, rather it's in support of a personal project. (I'm converting this project-- https://github.com/crspybits/SMSyncServer over to using Swift).

[shmuelk]

At one time I wanted KituraNet's ServerRequest to be a "Readable stream" as you put it, and for ServerResponse to be a "writable stream". In addition the underlying Socket class was also written like that, with the intention to support pipe like constructs.

Overtime, we started to see more and more of Foundation and we moved to using BlueSocket. The idea was sort of dropped.

Now I will point out that both Kitura's RouterRequest and KituraNet's ServerRequest classes support various read() functions that enable you to read in the body of the request. There is no need to write the body to disk. You could even use BodyParser to parse a multipart form body, if that was used to upload the file.

Lastly, at this time, KituraNet reads into memory the entire HTTP request. At one time it didn't do this, and during the course of some performance improvement work it was determined that simply reading in the entire request was more efficient. That may limit the size of files you can reasonably upload.

[crspybits]

Thanks for getting back to me, @shmuelk. I did see the read methods. Probably the most direct way of achieving my goal would be to just pass the RouterRequest to my Google Drive interface, and have it use the RouterRequest read method. Perhaps that's the way I should go for now. Simple and direct for now. And perhaps extend later. For right now that will tie my Google Drive interface to the RouterRequest. Maybe I'll make a light wrapper layer that will remove this dependency.

I have started work on a stream interface, and will likely continue tinkering on that (see https://github.com/crspybits/SwiftyStreams).

I saw your strategy of reading the entire HTTP request into memory. Not a problem for me for now, but for the kind of file sizes that systems like Google Drive can handle, it could be a problem in the future. I.e., in my system where I'm using my server in part as kind of redirector to pass data from a mobile client app up to different cloud storage systems.

One more thought: It would be great if eventually Kitura and other Swift-server side environments could adopt the same Streaming interface-- so as to improve interoperability. That might be asking for a miracle at this point however! :).