Add Multiple instances support

[ayashjorden]

Hi @pcfens, We ship most of our logs to Logstash for parsing and enrichment. As some of our logs are already formatted as JSON objects and do not need any modification, we can ship those lines directly to ES.

The dependencies for this feature are:

1. Install FileBeat 5.x (which introduced JSON parsing feature, among other features)
2. Support installing multiple instances of FileBeat on the same host.

The later is due because FileBeat itself doesn't have the ability to configure which prospector ships through which output.

You're more then welcome to take a look on this fork. the above bullets are implemented and working.

I'd be happy to get your take on the above :)

N.B. I still need to modify the tests to the new structure and solve some Docker issues preventing Beaker tests from running.

Thank you, Yarden

[pcfens]

I just released v5 support (though without JSON parsing settings - soon though, I'll probably grab code from your fork for that).

I hadn't thought of running multiple instances on a single host, but it makes sense. If we can add support for non-systemd systems to your fork then I think that model will work really well. I'm also trying to maintain puppet 3 compatibility for a little bit longer, hopefully at least until its official EoL at the end of the year.

[ayashjorden]

Hi @pcfens, What are the next steps? how can we move this forward?

[pcfens]

We need to get init/upstart/systemd templates set up, ideally with beaker tests. I like the approach you've used in your fork, we just need to expand it a bit.

I won't have the time to write it in the next few weeks, but am happy to help.

[acbox]

+1

[tombburnell]

+1 we have multiple outputs that we don't want delayed if one is down.

[KlavsKlavsen]

+1 for this. We also have some inputs which need to go through logstash and some who we'd like to just sent directly.

[rodolfosalgado]

It would be awesome to add multiple instances. Is there any news on this ?

[pcfens]

I think that it's a great idea - I just haven't had any time to work on it. I'm always open to PRs if someone else has time.

[ikkaro]

+1 for multiple instances, we need it to send items to different logstash servers (customer+we) and we don't want to install a local logstash for routing.

[weichuliu]

+1 for this

[matthagenbuch]

+1

[b-0-b]

Definitely +1

elastic-elasticsearch supports multiple instances. Maybe some inspiration could be taken from there :) https://github.com/elastic/puppet-elasticsearch/blob/master/manifests/instance.pp