axoflow
Move EOF detection and close-on-input to the proto
Fixes the collectd() destination (we never read the server's response).
Fixes possible starvation issue on the destination side when reading server responses
Fixes and workarounds TLS-related state handling issues
Fixes #604
@bazsi Please check the last patch, I could solve this layering problem without introducing additional buffers. I like this approach very much, it's clear in all layers what is happening (though I still consider it a workaround, which will probably stay with us forever).
@MrAnno As I could see in the changes, the handling of the IO conditions is correct, just this tiny piece is left out, without that, pipe destinations will close the connection continuously on any input detected on the bi-directionally opened pipes.
@HofiOne The fix suggested causes message loss as we'll read our own output, #665 should fix the issue.
@HofiOne The fix suggested causes message loss as we'll read our own output, #665 should fix the issue.
@MrAnno
Of course, not allowing IO in on a destination is a better choice than dropping the otherwise not wanted incoming, self written messages, but it is strange, I've run several tests, and dropping the input was well enough in this case, I've never seen a message loss.
I mean losing syslog-ng messages from the message queue and from the log path.
Reading the pipe directly can be misleading, depending on when and how you read it.
Did you check the produced output of a further-chained log path?
Just attach e.g. another pipe source to that pipe destination and a file destination to the source. I've sent thousands of messages in multiple scenarios, just like the kira tests do, and I could not see a message loss with flow control turned on.
The Python script I attached in my PR can show how syslog-ng would steal messages from the real destination if that destination is reading slowly.
A traditional pipe is not a broadcaster mechanism. If you have multiple readers, they will steal data from each other.
Further thinking of this, I thought dropping the self-written message from the self-read message pipe should not have effect on a secondarily attached pipe reader. I've used socat as a client to test this, also corrected the kira-tests, the tests both read the pipe directly with socat that writes its output to a file, and also have the pipe destination an attached source+destination I described above. All messages always arrived into the socat written file and into the secondary file destination at the end of the log path.
But yes, I've read it once again, it was long ago I used pipes directly, they can steel messages from each other, I was just lucky it seems.
After writing, syslog-ng goes back to its main loop and schedules a separate read job into its thread pool. That's probably enough delay for normal pipe readers to read the data before syslog-ng can do anything, that's why I needed to slow down my reproducer code with a sleep.
But, this means we should check that no more than one source is attached to a given pipe destination, and warn the user that message loss can happen if there are multiple readers are in the log path, but probably it can be a different fix