Retrieved chunks are not really HTTP chunks.

[vincentbernat]

Hi!

Because node.js HTTP parser doesn't expose callbacks to hook some actions when receiving chunks, it seems difficult to really detect HTTP chunks. On this matter, see this old PR of mine: https://github.com/joyent/http-parser/pull/162

You are likely to detect more chunks than real HTTP chunks. For example, when I try with google, I get this:

But looking at the capture in wireshark, I get this:

https://www.cloudshark.org/captures/0de5e8d5519b

So, the timing is accurate. I get the first chunk at 171ms. However, the first chunk is 0x8000 bytes (size is in packet 6). However, we see a pause at 177 ms. This is because the transmission window on the server is full. It waits for our acks. At 200ms, we resume and we can see in the capture that this is not a new chunk (packet 22 starts with "osition"). If we go at packet 58 which is the end of the second pause, we can see this is not the beginning of a chunk either (starts with ";").

The second chunk starts at packet 52 and its size is 0x6481 (25729). The chunk starts in the middle of a TCP packet. I think this is unlikely that you got the data callback triggered on this boundary (but that's difficult to know because it gets triggered a lot of time, maybe we could log the first few bytes to check).

We could go on with the analysis, but I would conclude that the pauses that you see do not have to be the boundary of an HTTP chunk (none of the two large pauses are) and you don't have to get a dot on an HTTP chunk boundary (but that's more difficult to check).

I don't think there is anything to do at this level since node doesn't expose "end of chunk". You'll have to rewrite an HTTP parser (or convince node guys to accept my PR ;-) ).

Please note that I base this in the state of the HTTP parser in the Joyent repository. I believe this is the same used by node but I didn't check if node.js parser has evolved or not.