Failure when running elasticsearch v.7.x

[palisadoes]

I'm having an issue with inserting an index when using elasticsearch v7.x, specifically elasticsearch-7.3.2-1.x86_64

It seems to be unrelated to the pmacct installation and solely related to the version of elasticsearch as I'm not relying on any pmacct output when getting this message. Please verify if possible.

Details

Prior to diverting netflow traffic to pmacct-to-elasticsearch using this command:

*/5 *	* * *     root  pmacct -l -p /var/spool/pmacct/netflow.pipe -s -O json -e | pmacct-to-elasticsearch netflow

I get the following error:

2019-09-27 21:37:12,549 ERROR Error while creating index netflow-2019-09-27: An error occurred while creating index netflow-2019-09-27 from template /etc/p2es/new-index-template.json: error unknown

It is specifically being caused by the pmacct-to-elasticsearch netflow pipe. When I curl the expected url I get the following error.

[root@tsv009 tmp]# curl http://localhost:9200/netflow-`date +%F`/socket/_search?pretty
{
  "error" : {
    "root_cause" : [
      {
        "type" : "index_not_found_exception",
        "reason" : "no such index [netflow-2019-09-27]",
        "resource.type" : "index_or_alias",
        "resource.id" : "netflow-2019-09-27",
        "index_uuid" : "_na_",
        "index" : "netflow-2019-09-27"
      }
    ],
    "type" : "index_not_found_exception",
    "reason" : "no such index [netflow-2019-09-27]",
    "resource.type" : "index_or_alias",
    "resource.id" : "netflow-2019-09-27",
    "index_uuid" : "_na_",
    "index" : "netflow-2019-09-27"
  },
  "status" : 404
}
[root@tsv009 tmp]

The new-index-template.json file I'm using is the one provided by your latest code:

{
  "mappings": {
    "_default_": {
      "_all" : { "enabled" : false },

      "_source" : { "enabled" : true },

      "dynamic_templates": [
        {
          "string_template" : {
            "match" : "*",
            "match_mapping_type" : "string",
            "mapping": { "type": "keyword", "index": true }
          }
        }
      ],

      "properties": {
        "@timestamp": { "type": "date" },
        "bytes": { "type": "long" },
        "packets": { "type": "long" },
        "flows": { "type": "long" }
      }
    }
  }
}

It seems to be unrelated to the pmacct installation and solely related to the version of elasticsearch as I'm not relying on any pmacct output when getting this message. Please verify if possible.

[palisadoes]

Elastic search is up and running as you can see from this output:

# curl localhost:9200/
{
  "name" : "node-001",
  "cluster_name" : "pmacct",
  "cluster_uuid" : "pA8zMKYST2e_iQ2XQEYR7w",
  "version" : {
    "number" : "7.3.2",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "1c1faf1",
    "build_date" : "2019-09-06T14:40:30.409026Z",
    "build_snapshot" : false,
    "lucene_version" : "8.1.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

[pierky]

I see something changed with ES7: https://www.elastic.co/guide/en/elasticsearch/reference/7.0/mapping.html

Trying to figure out how the new template should look like...

[pierky]

Hello @palisadoes, I've just released v0.3.3-alpha1 which should work with ES7 too. The ES_Type configuration setting should be left empty in order to achive compatibility with ES7. Also the template file to build the index has been changed.

Could you verify if it works for you now? To install it via pip, you probably need to pass the --pre argument: pip install --pre pmacct-to-elasticsearch==0.3.3a1

Thanks.