add field operation in external datasource

[scampi]

For example, get substring of a field @doc[_source][substr(fieldname,0,4)]@.

https://groups.google.com/a/siren.solutions/forum/#!topic/kibi-users/q4nj8m4yqQA

[szydan]

Maybe this format would be better
@substring(doc[_source][fieldname],0,4)@

[martienl]

For text fields, it seems to be necessary to use python for this.

So far I found a way to get the info:

• needs https://github.com/elastic/elasticsearch-lang-python
• in es-dir: bin/plugin install elasticsearch/elasticsearch-lang-python
• in elasticsearch.yml script.inline: true script.indexed: true

curl -XDELETE "http://localhost:9220/test"

curl -XPUT "http://localhost:9220/test/doc/1?refresh" -d' { "ipc": "B01D53/92" }'

curl -XPUT "http://localhost:9220/test/doc/2?refresh" -d' {
"ipc": "H01M008/12" }'

curl -XGET "http://localhost:9220/test/_search?pretty" -d' { "script_fields": { "ipc_short": { "script": "_source[selectfld][begin:end]", "lang": "python", "params" : {"selectfld": "ipc", "begin" : 0, "end": 4} }, "ipc_txt": { "script": "_source["ipc"]" } } }'

• in config/scripts/substring.py _source[selectfld][begin:end]
• doesn't work (yet)

curl -XGET "http://localhost:9220/test/_search?pretty" -d' { "script_fields": { "script : { "file" : "substring", "lang": "python", "params" : {"selectfld": "ipc", "begin" : 0, "end": 4} } }, "ipc_txt": { "script": "_source["ipc"]" } } }'

and now?

[scampi]

Thanks for reporting on what you did @martienl this was useful for me.

I think for the moment, you have to create another field in your index that contains the substring you are interested in. The scripted field would be very nice and quite powerful, but it is nowhere yet possible on two levels:

1. on the UI side, there is no way to manage a scripted field with lang python and with type different from numeric. I thought https://github.com/simianhacker/kibana-hacks could be a way around for a time but Kibi is not yet synced with Kibana 5.
2. on the management side of the external datasource which does not support the injection of scripted fields. For reference, this https://github.com/sirensolutions/kibi/blob/75d8098158f2cb0763b3f313f86ce13e633a185d/src/plugins/kibi_core/lib/query_helper.js#L83 needs to be updated to pass scripted fields.

I am keeping this issue open for keeping this feature in the plan, but a scripted field is not a solution for your issue yet.

[martienl]

Thanks, Scampi!