Add support for out-of-order audit messages

[mark-ignacio]

I've made a custom audit.Assembler to handle auditd messages at the end of a log pipeline that doesn't guarantee ordered delivery. This seems like a generally useful thing, so would you accept some PRs to get this added in?

I think this would mean refactoring libaudit.Reassembler to an interface and implementing a UnorderedReassembler struct that emits a slice of messages when it gets a complete set. FWIW, our NewUnorderedReassembler() has the same args as NewReassembler() and is used basically the same way.

I promise to also commit a bunch of tests at the same time. :)

[andrewkroh]

Hi Mark, I don't quite understand how UnorderedReassembler is different from the current Reassember. You could provide some more details or an example. I may have forgotten some of the intricacies of the reassember code. And if you would like to open a PR with what you have today as is I can take a look at that.

[mark-ignacio]

Sure - the short of this is that Reassembler marks a message as complete as soon as it encounters a few kinds of audit message types:

https://github.com/elastic/go-libaudit/blob/db7d4996da54bed87ecf047e6c19d7656d237082/reassembler.go#L187-L191

https://github.com/elastic/go-libaudit/blob/db7d4996da54bed87ecf047e6c19d7656d237082/reassembler.go#L266-L270

In an out-of-order world, these ending messages don't guarantee that this is a safe time to flush messages out of the assembler, so UnorderedReassembler keeps state about whether a group of messages satisfies a complete event before we can feed it to aucoalesce.CoalesceMessages. (e.g. an EXECVE message declares how many PATH messages it also has, so we need to get those before we can emit the event slice).

I'll try open a PR sometime this week with what we've got!

[Darkheir]

I would be interested to see the way you handle out of order messages. On some hosts I see it happening quite a lot and by pushing messages once auparse.AUDIT_PROCTITLE is received prevent the message from being handled.