transporter icon indicating copy to clipboard operation
transporter copied to clipboard

ElasticSearch child-parent relationship not work?

Open tolerious opened this issue 6 years ago • 4 comments

I have two collections, one named books and the other named bookchapters, books is parent of bookchapters, they both in db engineer I wrote the config file like this:

var source = mongodb({
  "uri": "mongodb://engineer:engineer@localhost:27017/engineer",
})

var sink = elasticsearch({
  "uri": "http://localhost:9200/engineer",
  "parent_id": "books_id"
})

t.Source("source", source, "/.*/").Save("sink", sink, "/.*/")

without doubt, before I run ./transpoter-0.5.1-darwin-amd64 run pipeline.js, I have configed the engineer index like this:

PUT /engineer
{
  "mappings":{
    "books":{},
    "bookchapters":{
      "_parent": {
        "type": "books"
      }
    }
  }
}

after that I run ./transpoter-0.5.1-darwin-amd64 run pipeline.js but I got errors like this:

INFO[0002] iterating...                                  collection=bookchapters
ERRO[0002] _bulk failed list [0]                         error="&elastic.ErrorDetails{Type:"routing_missing_exception", Reason:"routing is required for [engineer]/[bookchapters]/[59e85e58f18d06f0ddc178fa]", ResourceType:"", ResourceId:"", Index:"engineer", Phase:"", Grouped:false, CausedBy:map[string]interface {}(nil), RootCause:[]*elastic.ErrorDetails(nil), FailedShards:[]map[string]interface {}(nil)}" executionID=17 id=59e85e58f18d06f0ddc178fa index=engineer type=bookchapters version=5 writer=elasticsearch
ERRO[0002] _bulk failed list [1]                         error="&elastic.ErrorDetails{Type:"routing_missing_exception", Reason:"routing is required for [engineer]/[bookchapters]/[59e85e58f18d06f0ddc178fc]", ResourceType:"", ResourceId:"", Index:"engineer", Phase:"", Grouped:false, CausedBy:map[string]interface {}(nil), RootCause:[]*elastic.ErrorDetails(nil), FailedShards:[]map[string]interface {}(nil)}" executionID=17 id=59e85e58f18d06f0ddc178fc index=engineer type=bookchapters version=5 writer=elasticsearch
ERRO[0002] _bulk failed list [2]                         error="&elastic.ErrorDetails{Type:"routing_missing_exception", Reason:"routing is required for [engineer]/[bookchapters]/[59e85e59f18d06f0ddc178ff]", ResourceType:"", ResourceId:"", Index:"engineer", Phase:"", Grouped:false, CausedBy:map[string]interface {}(nil), RootCause:[]*elastic.ErrorDetails(nil), FailedShards:[]map[string]interface {}(nil)}" executionID=17 id=59e85e59f18d06f0ddc178ff index=engineer type=bookchapters version=5 writer=elasticsearch
ERRO[0002] _bulk failed list [3]                         error="&elastic.ErrorDetails{Type:"routing_missing_exception", Reason:"routing is required for [engineer]/[bookchapters]/[59e85e5af18d06f0ddc17902]", ResourceType:"", ResourceId:"", Index:"engineer", Phase:"", Grouped:false, CausedBy:map[string]interface {}(nil), RootCause:[]*elastic.ErrorDetails(nil), FailedShards:[]map[string]interface {}(nil)}" executionID=17 id=59e85e5af18d06f0ddc17902 index=engineer type=bookchapters version=5 writer=elasticsearch
ERRO[0002] _bulk failed list [4]                         error="&elastic.ErrorDetails{Type:"routing_missing_exception", Reason:"routing is required for [engineer]/[bookchapters]/[59e85e5bf18d06f0ddc17903]", ResourceType:"", ResourceId:"", Index:"engineer", Phase:"", Grouped:false, CausedBy:map[string]interface {}(nil), RootCause:[]*elastic.ErrorDetails(nil), FailedShards:[]map[string]interface {}(nil)}" executionID=17 id=59e85e5bf18d06f0ddc17903 index=engineer type=bookchapters version=5 writer=elasticsearch
ERRO[0002] _bulk failed list [5]                         error="&elastic.ErrorDetails{Type:"routing_missing_exception", Reason:"routing is required for [engineer]/[bookchapters]/[59e85e5df18d06f0ddc17907]", ResourceType:"", ResourceId:"", Index:"engineer", Phase:"", Grouped:false, CausedBy:map[string]interface {}(nil), RootCause:[]*elastic.ErrorDetails(nil), FailedShards:[]map[string]interface {}(nil)}" executionID=17 id=59e85e5df18d06f0ddc17907 index=engineer type=bookchapters version=5 writer=elasticsearch
ERRO[0002] _bulk failed list [6]                         error="&elastic.ErrorDetails{Type:"routing_missing_exception", Reason:"routing is required for [engineer]/[bookchapters]/[59e85e5df18d06f0ddc17908]", ResourceType:"", ResourceId:"", Index:"engineer", Phase:"", Grouped:false, CausedBy:map[string]interface {}(nil), RootCause:[]*elastic.ErrorDetails(nil), FailedShards:[]map[string]int

tolerious avatar Nov 30 '17 07:11 tolerious

What version of Elasticsearch is this?

codepope avatar Nov 30 '17 12:11 codepope

@codepope I use ElasticSearch 5.6.2

tolerious avatar Dec 01 '17 00:12 tolerious

I'm not super familiar with the parent/child feature in elasticsearch, maybe @johnjjung could comment as he worked on the implementation in transporter?

jipperinbham avatar Dec 12 '17 14:12 jipperinbham

@codepop @tolerious did you create the parent child relationship before you start the sync?

You must create parent child mappings before you start transporter

Also try running it with the debug flag so you can see more detailed errors

https://github.com/compose/transporter/tree/master/adaptor/elasticsearch#example-of-parent-child-mapping for example of how to create the mapping on elasticsearch

johnjjung avatar Dec 13 '17 23:12 johnjjung