Corrupt messages on streaming http_client disconnect

[nicktelford]

When in streaming mode, a if an http_client input is disconnected prematurely, it's possible that a prematurely truncated message will be processed.

Since in HTTP streams messages are either explicitly delimited by MIME multi-part headers, or by new-lines, it should be possible to detect when a message has been prematurely closed and prevent it from being processed in the downstream pipeline.

[Jeffail]

Will need a big of digging but this is probably going to need to be an option on the codec type, where on EOF we discard partial data rather than flush it.

[garbelini]

We are seeing this again. This time we compared this benthos configuration with curl and curl keeps up while benthos http client stream bails with ERRO Failed to read message: unexpected EOF

Of all the streams we have targeting the same system with the same configuration, the one that is failing is the one with the the lowest volume.

input:
  label: ""
  http_client:
        url: https://hostname
        verb: GET
        headers:
          Content-Type: application/json
        stream:
          enabled: true
        basic_auth:
            enabled: true
            username: ********
            password: ********