[jobreceiver] IO issues with logs entires

[c-kruse]

Ran into this a bit in development and now in testing.

We are using go's https://pkg.go.dev/os/exec#Cmd.StdoutPipe, which creates an os.Pipe. On linux that corresponds to a named pipe, which can be especially sensitive to read latency and errors on read after all of the writer fds are closed even when the small read buffer is not empty (resulting in dropped data.) When sensu runs a command it sets exec.Cmd.Stdout to a buffer, and the exec package behind the scenes creates an os.Pipe and starts a goroutine that aggressively reads that pipe into the buffer.

To avoid problems we should add a similar buffer somewhere. The event output handler effectively replicates that buffer here, but the log entries handler currently tries to process records in the read loop. I hope some form of buffer may make this "good enough", but but because of the flawed nature of this form of IPC we can't really apply effective backpressure like we're accustomed to. I suspect this issue exists in the installed collector script sources as well, and just gets papered over with scala the libraries.

[c-kruse]

Side note: I have my doubts around the suitability of os pipes, but they might not be the root of the problem. from Cmd.StdoutPipe:

Wait will close the pipe after seeing the command exit, so most callers need not close the pipe themselves. It is thus incorrect to call Wait before all reads from the pipe have completed.

May have just goofed on this and need to reorganize how the command package works to accommodate this.