Explore use case of event log forwarding

[scudette]

trafficstars

Functionally equivalent to winfilebeats, this use case is just about easily forwarding logs to the server. Some obvious advantage to winfilebeats include

1. Velociraptor uses standard http coms
2. Velociraptor can deliver data faster since data is sent in bulk json blobs
3. Client side file buffering can ensure the logs are forwarded even when offline, then uploaded when possible
4. Server side journal ensures that even
5. if elastic lags the events are queued in the server until the elastic server can catch up.
6. If velociraptor is already installed on the endpoint it's a pain to install yet another agent so this just simplifies the whole thing

This issue is about exploring this use case. What kind of functionality do we need? How do we prove performance?

[predictiple]

Taking one step back from the problem statement we have to ask: why one would want to do event forwarding? To my mind there are 2 reasons:

1. For archival purposes, i.e. so that events can be searched later. Usually this also implies centralisation,

2. Because better search capabilities are available elsewhere, rather than on the endpoint.

For the first point I believe there are many existing solutions available, including Microsoft's native ones for Windows event logs.

For the 2nd point, it's not not necessarily the case that VR lacks the desired capabilities. If VR is part of a data pipeline there may be complex enrichments and/or filtering that are applied further up the pipeline. Those enrichments may need to cross-reference data from other non-VR sources. Enrichments may also be reapplied on the data, for example geoip or malware lookups may have results that change from day to day and should therefore be "refreshed". In general enrichments are best done as close to the point of analysis as possible, unless they require other (perhaps transient) data from the endpoint. If we consider a use case such as checking a huge list of hashes or IPs then it doesn't seem like a good idea to apply that detection on the endpoint, even though it techically could be done (as demonstrated by the Windows.Forensics.LocalHashes.Usn artifact).

[scudette]

This is currently working pretty well with a very high event rate shipped to the server. Parsing and indexing events is out of scope for Velociraptor but we can at least collect the raw data.

Alerting is now evolving into its own framework using #2631 as an initial implementation.