Elastisearch 5 compatibility

[omelendres]

Hi,

Having some issue to make the script work with elasticsearch 5. For thesting purposes I am using the included the file mail.log

Here is the error log

node server.js

missing required arg PARSE ERROR for syslog: PARSE ERROR for syslog:Jul 30 00:10:11 imac27 madeup: Gobbely Gook end of /root/node_modules/log-ship-elastic-postfix/test/mail.log doQueue: 3 lines end of file watching /root/node_modules/log-ship-elastic-postfix/test/mail.log { Error: [parsing_exception] Unknown key for a START_OBJECT in [filter]., with { line=1 & col=11 } at respond (/root/node_modules/elasticsearch/src/lib/transport.js:289:15) at checkRespForFailure (/root/node_modules/elasticsearch/src/lib/transport.js:248:7) at HttpConnector. (/root/node_modules/elasticsearch/src/lib/connectors/http.js:164:7) at IncomingMessage.wrapper (/root/node_modules/lodash/lodash.js:4968:19) at emitNone (events.js:91:20) at IncomingMessage.emit (events.js:185:7) at endReadableNT (_stream_readable.js:974:12) at _combinedTickCallback (internal/process/next_tick.js:74:11) at process._tickCallback (internal/process/next_tick.js:98:9) status: 400, displayName: 'BadRequest', message: '[parsing_exception] Unknown key for a START_OBJECT in [filter]., with { line=1 & col=11 }', path: '/postfix-orphan*/postfix-orphan/_search', query: { size: 3072 }, body: '{"filter":{"terms":{"_id":["3mPVKl0Mhjz7sXv"]}}}', statusCode: 400, response: '{"error":{"root_cause":[{"type":"parsing_exception","reason":"Unknown key for a START_OBJECT in [filter].","line":1,"col":11}],"type":"parsing_exception","reason":"Unknown key for a START_OBJECT in [filter].","line":1,"col":11},"status":400}', toString: [Function], toJSON: [Function] }

Seems like the template in elasticsearch is not being created. I I try to create it manually I get the following errors:

curl -XPUT 'http://localhost:9200/_template/postfix-orphan' [email protected]

{"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"Failed to parse mapping [postfix-orphan]: Field [_id] is defined twice in [postfix-orphan]"}],"type":"mapper_parsing_exception","reason":"Failed to parse mapping [postfix-orphan]: Field [id] is defined twice in [postfix-orphan]","caused_by":{"type":"illegal_argument_exception","reason":"Field [id] is defined twice in [postfix-orphan]"}},"status":400}

[msimerson]

Field [_id] is defined twice in [postfix-orphan]

That's an interesting error. I haven't had time to upgrade to ES 5 (I tried it briefly, ran into a number of issues that weren't quickly resolvable, so I reverted for now). I may get to this but it's really low priority for me so it won't likely be soon.

PRs welcome.

[omelendres]

Seems like an issue with the new elasticsearch rules on field name characters. Will try to make some tests changing the template. Thanks

[omelendres]

Seems the issue is related to the field name _id. It is ok to rename it to just id.

[msimerson]

So if one renames _id to id, that resolves the issue?

Looks like a pretty quick and simple change:

[matt] ~/git/log-ship-elastic-postfix $ grep _id `gfind`
./index-map-template.json:                "_id":     { "type" : "string", "index": "not_analyzed", "store" : true },
./lib/logship.js:        terms: { _id: Object.keys(pfQids) }
./lib/logship.js:      var qid = res.hits.hits[i]._id;
./lib/logship.js:      _id: qid,

[jcbf]

In elasticsearch 5 , all field names starting with _ are ignored. _something is reserved for meta fields.

[msimerson]

Since I don't have an ES 5 server up to test with, can I get someone to test this and make sure it does The Right Stuff[TM]? You can install this branch using npm with this syntax:

npm install msimerson/log-ship-elastic-postfix#21-elasticsearch-5

[omelendres]

There ase some errors with elasticsearch. Looks like the script is doing some search on an index not created yet

/node_modules/log-ship-elastic-postfix# node server.js
{ Error: ENOENT: no such file or directory, open '/etc/log-ship-elastic-postfix.ini'
    at Object.fs.openSync (fs.js:558:18)
    at Object.fs.readFileSync (fs.js:468:33)
    at new loadConfig (/root/node_modules/log-ship-elastic-postfix/lib/config.js:28:21)
    at module.exports (/root/node_modules/log-ship-elastic-postfix/lib/config.js:38:10)
    at PostfixToElastic.loadConfig (/root/node_modules/log-ship-elastic-postfix/lib/logship.js:51:14)
    at new PostfixToElastic (/root/node_modules/log-ship-elastic-postfix/lib/logship.js:19:8)
    at Object.createShipper (/root/node_modules/log-ship-elastic-postfix/lib/logship.js:408:12)
    at Object.<anonymous> (/root/node_modules/log-ship-elastic-postfix/server.js:15:24)
    at Module._compile (module.js:571:32)
    at Object.Module._extensions..js (module.js:580:10)
  errno: -2,
  code: 'ENOENT',
  syscall: 'open',
  path: '/etc/log-ship-elastic-postfix.ini' }
missing required arg
PARSE ERROR for syslog:
PARSE ERROR for syslog:Jul 30 00:10:11 imac27 madeup: Gobbely Gook
end of /root/node_modules/log-ship-elastic-postfix/test/mail.log
doQueue: 3 lines
end of file
watching /root/node_modules/log-ship-elastic-postfix/test/mail.log
{ Error: [parsing_exception] Unknown key for a START_OBJECT in [filter]., with { line=1 & col=11 }
    at respond (/root/node_modules/elasticsearch/src/lib/transport.js:289:15)
    at checkRespForFailure (/root/node_modules/elasticsearch/src/lib/transport.js:248:7)
    at HttpConnector.<anonymous> (/root/node_modules/elasticsearch/src/lib/connectors/http.js:164:7)
    at IncomingMessage.wrapper (/root/node_modules/lodash/lodash.js:4968:19)
    at emitNone (events.js:91:20)
    at IncomingMessage.emit (events.js:186:7)
    at endReadableNT (_stream_readable.js:974:12)
    at _combinedTickCallback (internal/process/next_tick.js:74:11)
    at process._tickCallback (internal/process/next_tick.js:98:9)
  status: 400,
  displayName: 'BadRequest',
  message: '[parsing_exception] Unknown key for a START_OBJECT in [filter]., with { line=1 & col=11 }',
  path: '/postfix-orphan-2017-07-05%2Cpostfix-2017-07-05/_search',
  query: { ignore_unavailable: true, size: 3072 },
  body: '{"filter":{"terms":{"id":["3mPVKl0Mhjz7sXv"]}}}',
  statusCode: 400,
  response: '{"error":{"root_cause":[{"type":"parsing_exception","reason":"Unknown key for a START_OBJECT in [filter].","line":1,"col":11}],"type":"parsing_exception","reason":"Unknown key for a START_OBJECT in [filter].","line":1,"col":11},"status":400}',
  toString: [Function],
  toJSON: [Function] }
^C
SIGINT received (Ctrl-C)
starting graceful shutdown
queue is active
queue is active
queue is active
queue is active
queue is active
queue is active
queue is active
queue is active
queue is active
queue is active
queue is active
queue is active
queue is active
queue is active
queue is active
queue is active
queue is active
queue is active
queue is active
queue is active
queue is active
queue is active
queue is active
queue is active
queue is active
queue is active
queue is active
queue is active
queue is active
queue is active
queue is active
queue is active
queue is active
queue is active
queue is active

[msimerson]

  message: '[parsing_exception] Unknown key for a START_OBJECT in [filter]., with { line=1 & col=11 }',
  body: '{"filter":{"terms":{"id":["3mPVKl0Mhjz7sXv"]}}}',

So, the problem is in the filter. I'm not sure what to change it to. This will take reading the ES 5 docs to see what else has changed.

[omelendres]

Couple of issues worth to mention:

The code searches for field 'id', which is also described on the template file, but the record example references to field 'qid'. Looks like the serach fails because field 'id' is not found. The filed searched is stored as 'qid'.

Another issue found is a warning showing from Kibana dashboard, which is related to the field 'events' of type array. The filed array can oly contain objects of the same type according to the documentation: https://www.elastic.co/guide/en/elasticsearch/reference/current/array.html On the code there is a mixed type of date and string

[msimerson]

On the code there is a mixed type of date and string

I don't think so. The array events is populated with objects. At times the objects may have differing properties but they are still of the same time. (That's not saying Elasticsearch doesn't coherse the objects differently and thus produce Kibana warnings, but that's a different thing entirely, and has more to do with ES still handling arrays poorly).

[omelendres]

Don't know much about arrays on ES, haven't used them. How about the id search?

[msimerson]

The code searches for field 'id', which is also described on the template file, but the record example references to field 'qid'.

I think the record example had a typo. I've updated it to be just id.

[omelendres]

Thanks I'll post the results

[omelendres]

Another test, Elasticsearch throws an error on query string generated by nodejs path: '/postfix-orphan-2017-07-05%2Cpostfix-2017-07-05/_search'

Index not found

{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"postfix-2017-07-05","index_uuid":"na","index":"postfix-2017-07-05"}],"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"postfix-2017-07-05","index_uuid":"na","index":"postfix-2017-07-05"},"status":404}