rom-elasticsearch

rom-elasticsearch copied to clipboard

Bulk API is useful in cases of create or update multiple documents. Current API has some problems with performance in cases of inserting thousands of documents.

Bulk API

I use the following code in my project:

    class Bulk < ROM::Commands::Create
      relation :<my_relation_name>
      register_as :bulk

      # @api private
      def execute(tuples)
        body = tuples.flat_map { |tuple| to_body(input[tuple]) }
        dataset.client.bulk(body: body)
      end

      private

      def to_body(tuple)
        # [metadata(tuple), tuple]
        params = dataset.params
        [
          {
            index: {
              _index: params[:index],
              _type: params[:type],
              **({ _id: tuple.fetch(_id) } if _id).to_h
            }
          },
          tuple
        ]
      end

      # @api private
      def dataset
        relation.dataset
      end

      def _id
        relation.schema.primary_key_name
      end
    end

But I can't add it to the core, because if I add this to rom/elasticsearch/commands.rb it just doesn't work and I haven't found any places in the code where commands being configured.

Thanks for this code, managed to implement my bulk insert command using that. Probably you have already managed to configure this, I'll just leave the config needed here in case someone else is looking for same answer: config = ROM::Configuration.new(:elasticsearch, "http://#{host}:#{port}") config.register_command(Commands:: Bulk) #my commands are in commands namespace rom_es = ROM.container(config) ...

@pasivuorio You are welcome! I'm glad to see probably third person who is interested in rom-elasticsearch! 😅