filebeat: text search not working

[Morriz]

Hi, I have the logs streaming in the interface, and can select a specific pod and subcontainer, but when I just type a string (that I see in the logs) in the search bar and enter, it says No events found.

My configuration:

   {
      "index_patterns" : [
        {
          "es": {
            "default_index": "filebeat-*",
            "allow_url_parameter": false
          },
          "tail_interval_in_seconds": 2,
          "es_index_time_offset_in_seconds": 0,
          "display_timezone": "Etc/UTC",
          "display_timestamp_format": "YYYY MMM DD HH:mm:ss",
          "max_buckets": 500,
          "default_time_range_in_days" : 0,
          "max_hosts": 100,
          "max_events_to_keep_in_viewer": 5000,
          "nested_objects" : true,
          "fields" : {
            "mapping" : {
                "timestamp" : "@timestamp",
                "display_timestamp" : "@timestamp",
                "hostname" : "kubernetes.pod.name",
                "program": "kubernetes.container.name",
                "message": "log"
            }
          }
        }
      ]
    }

[Morriz]

The search works fine in Kibana btw.

Also, I don't see the logtrail search with the string landing in the kibana logs.

If I check the output of the web app's requests I see it passing this:

{searchText: "Finished", timestamp: null, rangeType: "gte", order: "asc", hostname: null,…}

Is that rangeType: "gte" ok? Or is it messing with the query?

[sivasamyk]

@Morriz My default logtrail searches in message field. Can you paste a sample document from elasticsearch?

[Morriz]

Sure. How?

[Morriz]

Hi @sivasamyk, can you give me instructions? Like I said, I can search with elasticsearch just fine.

[Morriz]

@sivasamyk I also want to point to my whole elk setup, so you can see the configs working together: https://github.com/Morriz/mostack/blob/master/charts/elk/templates/elk.yaml.

Am I missing something?

[sivasamyk]

curl '<Elastic_IP>:9200/filebeat-*/_search/?pretty' this should return documents from ES.

[Morriz]

Tnx, here's one:

      {
        "_index" : "filebeat-6.0.0-rc1-2017.12.04",
        "_type" : "doc",
        "_id" : "AWAgwKtj-b1YJGwQ6dO-",
        "_score" : 1.0,
        "_source" : {
          "@timestamp" : "2017-12-04T08:59:08.244Z",
          "source" : "/var/lib/docker/containers/6ba9b4114ecc202a5dd217cbd19d7f69e8296b0a022e937fffb5c37b793e613f/6ba9b4114ecc202a5dd217cbd19d7f69e8296b0a022e937fffb5c37b793e613f-json.log",
          "offset" : 5740,
          "log" : "I1204 08:58:43.651723       1 replicationcontroller.go:127] collected 0 replicationcontrollers",
          "stream" : "stderr",
          "time" : "2017-12-04T08:58:43.652055111Z",
          "prospector" : {
            "type" : "log"
          },
          "kubernetes" : {
            "pod" : {
              "name" : "kube-prometheus-exporter-kube-state-8fd545dfd-hk9bz"
            },
            "namespace" : "monitoring",
            "labels" : {
              "pod-template-hash" : "498101898",
              "release" : "kube-prometheus",
              "version" : "v1.1.0",
              "app" : "exporter-kube-state",
              "component" : "kube-state"
            },
            "container" : {
              "name" : "exporter-kube-state"
            }
          },
          "beat" : {
            "name" : "filebeat-bgcvz",
            "hostname" : "filebeat-bgcvz",
            "version" : "6.0.0-rc1"
          }
        }
      }

[sivasamyk]

@Morriz You mapping looks good for the given document. Can you let me know the following details:

1. Kibana and Logtrail version you are using
2. Elasticsearch version (I see it is 6.0.0)
3. Are you able to see all mapped fields in Logtrail UI
4. when you search via Kibana i assume you are using log:"Finished" as search text. Can you try the same text in Kibana also?

RangeType will used only when timestamp is not null.

[Morriz]

Sure:

1. Kibana and Logtrail both @ 5.6.3
2. Like you see: version 6
3. don't know if I can see all fields, but I attached a screenie
4. I did, and it works fine

[sivasamyk]

Can you try using log:"Finished" search text in logtrail search box? Logtrail version will be like 5.6.3-0.1.23

[Morriz]

Nothing changes: still nothing. I tried the lucene syntax before, no dice.

[sivasamyk]

@Morriz Your config and setup looks good to me. Let me try reproducing this with a local k8s installation.

[zytek]

Please check this as I believe you got hit by the same issue: https://github.com/elastic/beats/issues/5920

log field is of type keyword which doesn't behave the same as message field of type text that should be used for storing log lines.

[Morriz]

I just tried the new docker prospector and mapping "message": "log" works with that, but if I set the logtrail mapping to use "message": "message" it's empty...

So how do I configure it with type text?