sivasamyk
Support multiple filter types per index
By default most deployments of logstash I've seen have a single index logstash-date, the problem here is that LogTrail seems to expect you to create an index for every app or some sort of workaround.
It would be useful to match filters for example based on a program, e.g. if the program is nginx, then apply an nginx log filter, if it's kubes apply a kubes filter, syslog a syslog filter etc... rather than needing an separate index for each.
Please correct me if I'm wrong about how this works.
Can't you just type a query in the Logtrail search box?
E.g., type:syslog or whatever you've named the field.
It's not a problem with searching, it's a limitation of the Logtrail config file.
In all example configs you're only able to set one set of field mappings per logstash index. For example you might have nginx, kubernetes and syslog type logs in your logstash index, but the config only allows you to pick the field mappings for a single type of logs, e.g - you can't run both:
- https://github.com/sivasamyk/logtrail-config-examples/blob/master/syslog/logtrail.json and
- https://github.com/sivasamyk/logtrail-config-examples/blob/master/kubernetes/logtrail.json
On the same logstash index - which is what most people I have seen running logstash have and is the default logstash configuration to have a single logstash index per day.
Please do correct my if I'm interpreting the config file syntax incorrectly.
@sammcj Adding filter per program type will complicate the config structure and also currently program is flexible to be mapped to any field in ES document. Not always program will be restricted to list of pre-defined values. I personally prefer to route logs from diff app types (i.e assuming each has diff ES document schema) to separate indices.
I am experiencing the same limitation. I have indexes split by app but I usually want to query more apps at once to join related logs. It would be great to have multiple logtrail indexes using same es index pattern but different search, field mapping, output format, etc.
My environment are multiple Kubernetes clusters so classic per-host design doesn't apply to me. In my context host = cluster, program = pod name.
In my opinion only change that needs to be done is to add name key into config and use that in index pattern selectbox instead of mixing it with elasticsearch index name.
Something like this: https://github.com/fpytloun/logtrail/commit/68b3d4bcedf6becf7edabba65bcc4985c8b5cb42