Buffering requests/responses

[osxpinot]

Hello,

For my personal use, I've implemented buffering of requests and responses (including automatic gunzip) as a layer on top of this library. I don't think it is particularly useful to rewrite data at a chunk level, but admittedly good for performance.

I would like to do a pull request, but I'm at a loss on how to proceed and wanted to get some opinions and gauge interest.

Perhaps ctx.onRequestBody and ctx.onResponseBody?? This could possibly have weird interactions with .use middleware.

Let me know your thoughts

[joeferner]

If it is optional and doesn't cause too much code bloat I would be all for it

On Nov 28, 2016 12:52 AM, "osxpinot" [email protected] wrote:

Hello,

For my personal use, I've implemented buffering of requests and responses (including automatic gunzip) as a layer on top of this library. I don't think it is particularly useful to rewrite data at a chunk level, but admittedly good for performance.

I would like to do a pull request, but I'm at a loss on how to proceed and wanted to get some opinions and gauge interest.

Perhaps ctx.onRequestBody and ctx.onResponseBody?? This could possibly have weird interactions with .use middleware.

Let me know your thoughts

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/joeferner/node-http-mitm-proxy/issues/102, or mute the thread https://github.com/notifications/unsubscribe-auth/AAxXmZxrXsqtx-Ao1FzyFUyYkUQkbFkTks5rCmwggaJpZM4K9bZ5 .

[mk-pmb]

This sounds interesting. Could you explain more what you mean with "buffering of requests and responses"? Is it to delay them so you can interactively decide whether to edit them?

[jvilk]

Right now, the library is set up so that you can only modify chunks of a server response (via onResponseData). This is not too useful if you want to rewrite HTML, for example, because you need the entire HTML document before you can parse it.

With buffering, http-mitm-proxy would collect all of the server response chunks, concatenate them together, and present them to interception code for modification. The interception code would then provide a replacement server response to http-mitm-proxy for forwarding to the client.

Without this type of buffering, this library is not useful to me. :-(

[mk-pmb]

I see, so it's yes for delay and no for interactive.

would collect all of the server response chunks

Have you considered reducing the delay by scanning a sliding window? E.g. if you're looking for a script tag that you want to replace and the original page doesn't use fancy defenses against this, you can initially always forward all but the last 7 bytes of chunks that don't contain the string <script. Each time you find that string, you could extend the buffer window up to the next > to check the src attribute etc., and once all patches are applied, you can even disable buffering for this stream. Keeping the buffer size to a minimum gets even more useful if you want to live patch some few bytes in a large binary file or a video stream.

[jvilk]

While your suggestion would be helpful for many folks, and I appreciate that you are trying to help, a sliding window will not work for me. I'm using parsers that require the entire HTML/JS document, and what I am doing requires the whole document.

Right now, I'm using mitmproxy to talk to my Node.js code via a websocket. Efficient? No; not at all. Works? Yes.

[wisec]

@jvilk What about https://github.com/joeferner/node-http-mitm-proxy/blob/master/examples/processFullResponseBody.js ?

[jvilk]

I hadn't seen that! Nice. Do you know if the headers/status code can still be mutated at the end?