embulk
multi-fields config
Hi,
I like to push string input from CSV columns both as analyzed field for full-text search, and well as a not_analyzed field to Elasticsearch. Normally that can be done with the fields mapping parameter; https://www.elastic.co/guide/en/elasticsearch/reference/current/multi-fields.html How do I specify that using embulk ?
Hi. Currently, embulk-output-elasticsearch doesn't support index_template.
But I think it's a good idea to support this feature like fluent-plugin-elasticsearch is doing.
https://www.elastic.co/guide/en/elasticsearch/client/java-api/current/java-admin-indices.html#java-admin-indices-put-mapping
@jmwiersma @sakama How can we specify this mapping as plugin config options? Any thoughts?
Sorry, I feel a technical reply is beyond my abilities on this one.
For now as a work around I have pushed a template to ES, to created non_analyzed fields for any index named embulk-*. This is a mirror of how LogStash handles this. Example:
curl -XPUT http://localhost:9200/_template/embulk-* -d '{ "template" : "embulk-*", "settings" : { "index.refresh_interval" : "5s" }, "mappings" : { "_default_" : { "_all" : {"enabled" : true, "omit_norms" : true}, "dynamic_templates" : [ { "message_field" : { "match" : "message", "match_mapping_type" : "string", "mapping" : { "type" : "string", "index" : "analyzed", "omit_norms" : true, "fielddata" : { "format" : "disabled" } } } }, { "string_fields" : { "match" : "*", "match_mapping_type" : "string", "mapping" : { "type" : "string", "index" : "analyzed", "omit_norms" : true, "fielddata" : { "format" : "disabled" }, "fields" : { "raw" : {"type": "string", "index" : "not_analyzed", "doc_values" : true, "ignore_above" : 256} } } } } ] } } }'
@muga @jmwiersma
How can we specify this mapping as plugin config options? Any thoughts?
I think specify as a file is better. Description method of index template is too complex to specify as column option of embulk.
For example, fluent-plugin-elasticseach has 2 options.
Thank you. name and file path (or json content) are good as config options. Will take it.
