The async Logstash client does not support the "ttl" option

[mbudge]

Hi,

Any idea when TTL will be supported with the async client?

"The ttl option is not yet supported on an asynchronous Logstash client (one with the pipelining option set)"

https://www.elastic.co/guide/en/fleet/current/logstash-output.html

Thanks

[pierrehilbert]

@nimarezainia could you have a look here please?

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[nimarezainia]

@mbudge you should be able to use those configurations in the advanced yaml section of the Logstash. I believe the options should be available. @cmacknz please keep me honest.

we can look at adding the UI to the output to configure these extra options.

[cmacknz]

It isn't supported in the output itself, it's not a matter of being able to provide the configuration or not.

https://github.com/elastic/beats/blob/8f8f3137f2ed1a78a9db4d48831d6578831619f8/libbeat/outputs/logstash/async.go#L76-L79

[cmacknz]

The documentation on the TTL option from https://www.elastic.co/guide/en/beats/filebeat/current/logstash-output.html#_ttl:

Time to live for a connection to Logstash after which the connection will be re-established. Useful when Logstash hosts represent load balancers. Since the connections to Logstash hosts are sticky, operating behind load balancers can lead to uneven load distribution between the instances. Specifying a TTL on the connection allows to achieve equal connection distribution between the instances. Specifying a TTL of 0 will disable this feature.

The default value is 0. This setting accepts duration data type values.

[nimarezainia]

Yeah apologies. @mbudge wrote that also in the description, totally missed it.

[ar3diu]

So... is it just an error message and the TTL should be applied even if pipelining is enabled? Or the TTL setting is simply not supported in Logstash Output?

[cmacknz]

The TTL setting is ignored and does nothing when in async mode. In sync mode it works as described.

[ar3diu]

The TTL setting is ignored and does nothing when in async mode. In sync mode it works as described.

And how should the async mode be disabled? pipelining: 0?

[cmacknz]

Yes per https://www.elastic.co/guide/en/beats/filebeat/current/logstash-output.html#_pipelining

https://github.com/elastic/beats/blob/a4f576f34928669b821a0243af00b35d8d74918e/libbeat/outputs/logstash/logstash.go#L75-L79

[ar3diu]

I tried to set the following for the Logstash output in Fleet, but I still get some errors in regards to agent connections to the Logstash hosts. Any idea what could cause them?

pipelining: 0
ttl: 30s

error closing connection to logstash host <redacted>: tls: failed to send closeNotify alert (but connection was closed anyway): write tcp <redacted>-><redacted>: wsasend: An established connection was aborted by the software in your host machine., reconnecting...

[daniel-vasek]

I think the answer has not been answered. Setting the TTL is the only option how to evenly distribute the load through the load balancer (btw it's the default way how to load balance in Kubernetes). Setting pipelining to 0 reduces the throughput of the logging pipeline significantly, so the possibility of using TTL with pipelining would be super useful for those who need to deliver huge number of logs.