pmacct-to-elasticsearch
pmacct-to-elasticsearch copied to clipboard
Failure when running elasticsearch v.7.x
I'm having an issue with inserting an index when using elasticsearch v7.x, specifically elasticsearch-7.3.2-1.x86_64
It seems to be unrelated to the pmacct
installation and solely related to the version of elasticsearch
as I'm not relying on any pmacct
output when getting this message. Please verify if possible.
Details
Prior to diverting netflow traffic to pmacct-to-elasticsearch
using this command:
*/5 * * * * root pmacct -l -p /var/spool/pmacct/netflow.pipe -s -O json -e | pmacct-to-elasticsearch netflow
I get the following error:
2019-09-27 21:37:12,549 ERROR Error while creating index netflow-2019-09-27: An error occurred while creating index netflow-2019-09-27 from template /etc/p2es/new-index-template.json: error unknown
It is specifically being caused by the pmacct-to-elasticsearch netflow
pipe. When I curl the expected url I get the following error.
[root@tsv009 tmp]# curl http://localhost:9200/netflow-`date +%F`/socket/_search?pretty
{
"error" : {
"root_cause" : [
{
"type" : "index_not_found_exception",
"reason" : "no such index [netflow-2019-09-27]",
"resource.type" : "index_or_alias",
"resource.id" : "netflow-2019-09-27",
"index_uuid" : "_na_",
"index" : "netflow-2019-09-27"
}
],
"type" : "index_not_found_exception",
"reason" : "no such index [netflow-2019-09-27]",
"resource.type" : "index_or_alias",
"resource.id" : "netflow-2019-09-27",
"index_uuid" : "_na_",
"index" : "netflow-2019-09-27"
},
"status" : 404
}
[root@tsv009 tmp]
The new-index-template.json
file I'm using is the one provided by your latest code:
{
"mappings": {
"_default_": {
"_all" : { "enabled" : false },
"_source" : { "enabled" : true },
"dynamic_templates": [
{
"string_template" : {
"match" : "*",
"match_mapping_type" : "string",
"mapping": { "type": "keyword", "index": true }
}
}
],
"properties": {
"@timestamp": { "type": "date" },
"bytes": { "type": "long" },
"packets": { "type": "long" },
"flows": { "type": "long" }
}
}
}
}
It seems to be unrelated to the pmacct
installation and solely related to the version of elasticsearch
as I'm not relying on any pmacct
output when getting this message. Please verify if possible.
Elastic search is up and running as you can see from this output:
# curl localhost:9200/
{
"name" : "node-001",
"cluster_name" : "pmacct",
"cluster_uuid" : "pA8zMKYST2e_iQ2XQEYR7w",
"version" : {
"number" : "7.3.2",
"build_flavor" : "default",
"build_type" : "rpm",
"build_hash" : "1c1faf1",
"build_date" : "2019-09-06T14:40:30.409026Z",
"build_snapshot" : false,
"lucene_version" : "8.1.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
I see something changed with ES7: https://www.elastic.co/guide/en/elasticsearch/reference/7.0/mapping.html
Trying to figure out how the new template should look like...
Hello @palisadoes, I've just released v0.3.3-alpha1 which should work with ES7 too.
The ES_Type
configuration setting should be left empty in order to achive compatibility with ES7. Also the template file to build the index has been changed.
Could you verify if it works for you now? To install it via pip
, you probably need to pass the --pre
argument: pip install --pre pmacct-to-elasticsearch==0.3.3a1
Thanks.