Question: index mapping creation

[Eomm]

trafficstars

I'm sending to elasticsearch an http-request trace with the body. The index created by default will index all the body's fields and my server can't afford all that processing.. moreover, I'm not interested in index all the body fields, they should be just stored (I use other fields for search).

Since I'm using the index: 'pino%{DATE}' pattern, the index is created at runtime by this line if I'm not wrong:

https://github.com/pinojs/pino-elasticsearch/blob/56f132908cc93facdf45d5f7e658610be46d20f2/lib.js#L106

So, I'm not sure if I should configure the mapping of the desired index in this module with an additional feature, or there are other tricks in the elasticseach side. Could you give me some suggestion? Thanks

[mcollina]

If you customize that mapping in ES, is that retained on restart of pino-elasticsearch? Il 16 gen 2020, 11:08 +0100, Manuel Spigolon [email protected], ha scritto:

I'm sending to elasticsearch an http-request trace with the body. The index created by default will index all the body's fields and my server can't afford all that processing.. moreover, I'm not interested in index all the body fields, they should be just stored (I use other fields for search). Since I'm using the index: 'pino%{DATE}' pattern, the index is created at runtime by this line if I'm not wrong: https://github.com/pinojs/pino-elasticsearch/blob/56f132908cc93facdf45d5f7e658610be46d20f2/lib.js#L106 So, I'm not sure if I should configure the mapping of the desired index in this module with an additional feature, or there are other tricks in the elasticseach side. Could you give me some suggestion? Thanks — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[Eomm]

If you customize that mapping in ES, is that retained on restart of pino-elasticsearch?

Yes, if I create manually an index and then send the logs there, the index ignore the Request key:

{
  "mappings": {
    "log": {
      "properties": {
        "App": {
          "type": "text"
        },
        "Headers": {
          "type": "nested",
          "enabled": false
        },
        "Request": {
          "type": "nested",
          "enabled": false
        }
      }
    }
  }
}

[mcollina]

Would a flag to avoid adding the mapping be enough then?

[Eomm]

I had the chance to understand better the case:

• the code I linked send the logs to elastic, it doesn't create the index
• elastic, cause doesn't have the index of the day, create a new index with the default settings and it will index all the things

The logic I would apply in this module would be: if the user specified a custom mapping

• at startup create the index and ignore the error index already create
• if the user define the %{DATE} index's name, schedule a timeout to create the next index (or "when the generated index name has changed, create it then continue logging")

Am I the only one that would like to apply a different mapping than the default? 🤔 😅

[mcollina]

After getting to know ElasticSearch mappings, I would as well. Go ahead with the PR. Il 22 gen 2020, 22:02 +0100, Manuel Spigolon [email protected], ha scritto:

I had the chance to understand better the case:

• the code I linked send the logs to elastic, it doesn't create the index • elastic, cause doesn't have the index of the day, create a new index with the default settings and it will index all the things

The logic I would apply in this module would be: if the user specified a custom mapping

• at startup create the index and ignore the error index already create • if the user define the %{DATE} index's name, schedule a timeout to create the next index (or "when the generated index name has changed, create it then continue logging")

Am I the only one that would like to apply a different mapping than the default? 🤔 😅 — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.