elasticsearch-elixir icon indicating copy to clipboard operation
elasticsearch-elixir copied to clipboard

Add opts + action_type to Bulk.encode(!)/4

Open seanwash opened this issue 6 years ago • 3 comments
trafficstars

This PR includes a change to Bulk.encode/3 that adds a new opts param so that users can pass an action type besides "create", like "index", that can be used for bulk updating documents.

I know that there's a 2.0 rewrite happening that should expose more of the Bulk API, but this is something that we needed right now as we get rate limited by our Elasticsearch provider because they don't support concurrent index updates.

More context: https://github.com/danielberkompas/elasticsearch-elixir/issues/64

Example usage:

cluster = Dk.Elasticsearch.Cluster
config = Elasticsearch.Cluster.Config.get(cluster)
action_type = "index"

items =
  structs
  |> Dk.Repo.preload(document_preloads())
  |> Enum.map(
    &Elasticsearch.Index.Bulk.encode!(cluster, &1, index_name(), action_type: action_type)
  )

Elasticsearch.put( config, "/#{index_name()}/_doc/_bulk", Enum.join(items))
|> case do
  {:error, exception} ->
    Dk.Elasticsearch.send_elasticsearch_exception_notification(
      exception,
      %{action_type: action_type, caller: "Dk.Elasticsearch.Products.put_documents/1"}
    )

    {:error, exception}

  {:ok, response} ->
    {:ok, response}
end

cc @silviurosu

seanwash avatar Apr 16 '19 16:04 seanwash

Is there a status update on this? We also would like to use this functionality. Is there overlap with https://github.com/danielberkompas/elasticsearch-elixir/pull/78?

erikreedstrom avatar Oct 23 '19 12:10 erikreedstrom

@erikreedstrom It seems like there might be some small overlap, but if #78 does the job then I'd be happy to close this PR and contribute to your PR. I think this PR is failing the check only because of formatting. I ran mix format before pushing so I'm not exactly sure how to resolve that.

seanwash avatar Oct 23 '19 14:10 seanwash

#78 isn't our PR, but we have need to expand on the idea. Specifically for delete actions. We want to ensure we're moving forward with one that will be utilized.

erikreedstrom avatar Oct 23 '19 16:10 erikreedstrom

Closing in favor of #78

danielberkompas avatar Oct 01 '22 13:10 danielberkompas