Chunked Encoding (Streaming) Example

[johnjelinek]

Does cljs-ajax support comet-style http streaming via chunked-encoding?

[JulianBirch]

Not as yet, although it would be possible. I'm curious as to why you'd want to do this rather than use a websocket, though.

On Friday, April 17, 2015, John Jelinek IV [email protected] wrote:

Does cljs-ajax support comet-style http streaming via chunked-encoding?

— Reply to this email directly or view it on GitHub https://github.com/JulianBirch/cljs-ajax/issues/76.

Sent from an iPhone, please excuse brevity and typos.

[johnjelinek]

I'm depending on a streaming webapi and don't want to build an intermediary server to convert the stream into a websocket. Also, I don't need full-duplex communication.

— Sent from Mailbox

On Thu, Apr 16, 2015 at 11:07 PM, Julian Birch [email protected] wrote:

Not as yet, although it would be possible. I'm curious as to why you'd want to do this rather than use a websocket, though. On Friday, April 17, 2015, John Jelinek IV [email protected] wrote:

Does cljs-ajax support comet-style http streaming via chunked-encoding?

— Reply to this email directly or view it on GitHub https://github.com/JulianBirch/cljs-ajax/issues/76.

Sent from an iPhone, please excuse brevity and typos.

Reply to this email directly or view it on GitHub: https://github.com/JulianBirch/cljs-ajax/issues/76#issuecomment-93889087

[JulianBirch]

Makes sense. What sort of API would you expect in these circumstance (aka lets design it and see if it'll work :))

On Friday, April 17, 2015, John Jelinek IV [email protected] wrote:

I'm depending on a streaming webapi and don't want to build an intermediary server to convert the stream into a websocket. Also, I don't need full-duplex communication.

— Sent from Mailbox

On Thu, Apr 16, 2015 at 11:07 PM, Julian Birch <[email protected] javascript:_e(%7B%7D,'cvml','[email protected]');> wrote:

Not as yet, although it would be possible. I'm curious as to why you'd want to do this rather than use a websocket, though. On Friday, April 17, 2015, John Jelinek IV <[email protected] javascript:_e(%7B%7D,'cvml','[email protected]');> wrote:

Does cljs-ajax support comet-style http streaming via chunked-encoding?

— Reply to this email directly or view it on GitHub https://github.com/JulianBirch/cljs-ajax/issues/76.

Sent from an iPhone, please excuse brevity and typos.

Reply to this email directly or view it on GitHub: https://github.com/JulianBirch/cljs-ajax/issues/76#issuecomment-93889087

— Reply to this email directly or view it on GitHub https://github.com/JulianBirch/cljs-ajax/issues/76#issuecomment-93891225 .

Sent from an iPhone, please excuse brevity and typos.

[johnjelinek]

(streaming-ajax url (fn [chunk] (println (parse-json-to-map chunk))) "GET")

Maybe something like this, but you'd need to see how that plays well with core.async.

— Sent from Mailbox

On Thu, Apr 16, 2015 at 11:33 PM, Julian Birch [email protected] wrote:

Makes sense. What sort of API would you expect in these circumstance (aka lets design it and see if it'll work :)) On Friday, April 17, 2015, John Jelinek IV [email protected] wrote:

I'm depending on a streaming webapi and don't want to build an intermediary server to convert the stream into a websocket. Also, I don't need full-duplex communication.

— Sent from Mailbox

On Thu, Apr 16, 2015 at 11:07 PM, Julian Birch <[email protected] javascript:_e(%7B%7D,'cvml','[email protected]');> wrote:

Not as yet, although it would be possible. I'm curious as to why you'd want to do this rather than use a websocket, though. On Friday, April 17, 2015, John Jelinek IV <[email protected] javascript:_e(%7B%7D,'cvml','[email protected]');> wrote:

Does cljs-ajax support comet-style http streaming via chunked-encoding?

— Reply to this email directly or view it on GitHub https://github.com/JulianBirch/cljs-ajax/issues/76.

Sent from an iPhone, please excuse brevity and typos.

Reply to this email directly or view it on GitHub: https://github.com/JulianBirch/cljs-ajax/issues/76#issuecomment-93889087

— Reply to this email directly or view it on GitHub https://github.com/JulianBirch/cljs-ajax/issues/76#issuecomment-93891225 .

Sent from an iPhone, please excuse brevity and typos.

Reply to this email directly or view it on GitHub: https://github.com/JulianBirch/cljs-ajax/issues/76#issuecomment-93891309

[JulianBirch]

The thing that's holding me up on this is that I don't have any clojure code to use to test this. I don't suppose you have an example that would work?

[cvillecsteele]

+1111 on this... I need long poll also...

[JulianBirch]

It remains a good idea, and definitely within the remit of the project. I still need someone to come up with some httpkit code I can test against, however. :)

Just something that goes Hello CHUNK Polling CHUNK World will do the trick.

On Tuesday, 26 January 2016, Colin Steele [email protected] wrote:

+1111 on this... I need long poll also...

— Reply to this email directly or view it on GitHub https://github.com/JulianBirch/cljs-ajax/issues/76#issuecomment-175071362 .

Sent from an iPhone, please excuse brevity and typos.

[RadicalZephyr]

Hi @JulianBirch, there's a pretty good example of chunked encoding in the httpkit docs here: http://www.http-kit.org/server.html#async

Here's the same example without the implicit use clauses implied by the docs page:

(ns chunked.test
  (:require
    [org.httpkit.server :as s]
    [org.httpkit.timer :as t]))

(defn handler [request]
  (s/with-channel request channel
    (s/on-close channel (fn [status] (println "channel closed, " status)))
    (s/send! channel
             {:status 200
              :headers {"Content-Type" "text/html"}}
             false)
    (loop [id 0]
      (when (< id 10)
        (t/schedule-task (* id 200) ;; send a message every 200ms
                         (s/send! channel (str "message from server #" id) false))
        (recur (inc id))))
    (t/schedule-task 10000 (s/close channel))))

(s/run-server handler {:port 9090})

[Folcon]

@JulianBirch Not sure if you're still interesting in capturing incremental downloads from a streaming http response? I've gotten a version working with goog.net.XhrIo or is that becoming deprecated entirely? Not certain how good the process is, but it seems to work. I'm happy to try and work out some kind of minimal changeset if you're interested?

It's primarily based around hooking into here:

(when (fn? progress-handler)
      (doto this
        (.setProgressEventsEnabled true)
        (events/listen goog.net.EventType.PROGRESS progress-handler)))

It might be worthwhile having different ones for EventType.PROGRESS EventType.DOWNLOAD_PROGRESS EventType.UPLOAD_PROGRESS.

:progress-handler (fn [e] (let [resp-text (.getResponseText (.-currentTarget e))]
                                resp-text))

You can then do something with the partial result, in my case I did some chucked streaming of edn. Bit rough and ready, but it works =)...

[JulianBirch]

Hi, I'm super interested in getting this formalized. There's some obstacles, though, so I'll talk you through where I'm up to in terms of thinking about the problem:

From a different issue, I wrote:

OK, I'm pushing sorting this out properly until 0.8. The existing code will go into 0.7, but it's very XhrIo specific and plain won't work with a Java implementation. I think we're going to need two handlers: upload-progress and download-progress. The callback methods should probably take two parameters: byte counts current and total. total will be null if we can't work it out on download.

Since writing that, I've realised that download-progress also needs a third parameter: the decoded content. (Also, if you have a download progress handler your "final" handler shouldn't have content.)

However, the biggest obstacle is getting some decent integration tests running. I really need to fire up a server and run tests against it (I've got this for Java but not JavaScript). @RadicalZephyr posted some code to this earlier up.

So, I'll happily take any improvements you've got, but to get it as a supported feature I really need the working integration tests.

[Folcon]

@JulianBirch That's great, I've only got it working in ClojureScript land =)...

What kind of tests are you looking for? I've been treating my side pretty shabbily as the system I've built is pretty resilient already to being given bad data, so if it doesn't recognise something it discards it, and it's ok getting repeated entries, so I just dump it in and it gets handled by my downstream code.

I'll be tuning this up later, but it's on the order of months from now...

In terms of decoded content, that get's output from here:

:progress-handler (fn [e] (let [resp-text (.getResponseText (.-currentTarget e))]
                                resp-text))

From my perspective, just being able to hook in here with either goog.net.EventType: PROGRESS, UPLOAD_PROGRESS, or DOWNLOAD_PROGRESS and a function to apply against it solves a host of issues. Where if any of those event functions are defined, trigger (.setProgressEventsEnabled true).

I admit however that I haven't got as clear a view as you do about the overall architecture of the library, so you may have a more elegant approach than "did the user define one of": :progress-handler, :upload-progress-handler, or :download-progress-handler and is the handler a function?

So what do you need from me? I'm happy to chime back with what I can help with.

[JulianBirch]

Okay, the really basic thing that would help me a lot is an addition to the travis CI build that a) starts up http-kit and creates an endpoint called "/chunked". There's an existing http-kit bit in the tests, and it would be nice to only have the one, but I'll happily take it either way.

The end point does the following:

• Sends {:a 1}
• Sends {:b 2}
• Sends {:c 3}
• Closes

Then on the other side, a piece of code that connects up and verifies it receives that data in the progress handler.

I'm thinking :download-progress-handler should be a function that takes three parameters:

• count of bytes so far
• total count of bytes (if Content-Length was set, null if not)
• the translated content that was just received

I'm not sure we want :progress-handler on its own: it strikes me as just complicating the API for the sake of a minority of users's convenience.

:upload-progress-handler should probably just be a function taking two parameters

• count of bytes so far
• total count of bytes (always known this time)

Is this any help?

[Folcon]

So it is and it isn't :)...

That sounds great, however I'm not certain that the download-progress-handler would work as described? At least if it's based on goog.net.EventType.DOWNLOAD_PROGRESS, it might be possible, but so far I've not had much success getting the content just received, it's why I ended up using goog.net.EventType.PROGRESS. This might have been done for performance reasons, or equally I might have missed something. I can definitely give it a shot to confirm though.

Regarding the test case, you want something similar to this I assume? If so I'm happy to put a first pass together that if you're ok to review, I can adjust :)...

[JulianBirch]

Not sure. I think it’s possible with XmlHttpRequest but I could be wrong. (I am planning to make that the default one of these days.)

Yes, that’s the kind of thing. It’s harder to get working when not everything is in Java, sadly.

Looking forward to it

On Fri, 2 Nov 2018 at 19:46, Folcon [email protected] wrote:

So it is and it isn't :)...

That sounds great, however I'm not certain that the download-progress-handler would work as described? At least if it's based on goog.net.EventType.DOWNLOAD_PROGRESS, it might be possible, but so far I've not had much success getting the content just received, it's why I ended up using goog.net.EventType.PROGRESS. This might have been done for performance reasons, or equally I might have missed something. I can definitely give it a shot to confirm though.

Regarding the test case, you want something similar to this I assume https://github.com/JulianBirch/cljs-ajax/blob/master/test/ajax/test/server_interaction.clj#L32? If so I'm happy to put a first pass together that if you're ok to review, I can adjust :)...

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/JulianBirch/cljs-ajax/issues/76#issuecomment-435473987, or mute the thread https://github.com/notifications/unsubscribe-auth/AAFIkwkb3eaPrBem5PT7phbKPb5ky-zrks5urKEIgaJpZM4ECUqu .

-- Sent from an iPhone, please excuse brevity and typos.