feat(core): Log response bodies in HTTP request debug #15911

[jonathanpv]

Context

We are currently trying to implement previewing the response bodies from HTTP requests to help users debug Currently a draft PR to be reviewed by Bruce to see how to improve and move forward from this iteration Closes https://github.com/vectordotdev/vector/issues/15911

Solution

We're resorting to using the map_data combinator offered by the http crate and then creating a closure to extract the data and log it once the buffer is full.

Testing

vector.toml

[sources.in]
type = "http_client"
endpoint = "https://jsonplaceholder.typicode.com/todos/1"
scrape_interval_secs = 15

[sinks.out]
inputs = ["in"]
type = "console"
encoding.codec = "text"

Then run this command in the terminal

VECTOR_LOG=debug cargo run -- --config=./vector.toml

The output will look something like this, notice the HttpBodyDataReceived buffer showcases the first 20 bytes. This is hardcoded and we plan to make this configurable in the future.

023-01-25T21:21:53.720823Z DEBUG source{component_kind="source" component_id=in component_type=http_client component_name=in}: vector::internal_events::http_client: HttpBodyDataReceived buffer="[123, 10, 32, 32, 34, 117, 115, 101, 114, 73, 100, 34, 58, 32, 49, 44, 10, 32, 32, 34]"
2023-01-25T21:21:53.720961Z DEBUG source{component_kind="source" component_id=in component_type=http_client component_name=in}: vector::internal_events::common: Request completed.
{
  "userId": 1,
  "id": 1,
  "title": "delectus aut autem",
  "completed": false
}
2023-01-25T21:21:57.972230Z DEBUG sink{component_kind="sink" component_id=out component_type=console component_name=out}: vector::utilization: utilization=0.013923491211099079
2023-01-25T21:22:02.972407Z DEBUG sink{component_kind="sink" component_id=out component_type=console component_name=out}: vector::utilization: utilization=0.001392416617917251