elasticsearch-perl icon indicating copy to clipboard operation
elasticsearch-perl copied to clipboard

late flushing

Open Skeeve opened this issue 4 years ago • 0 comments

I think, this is wrong.

https://github.com/elastic/elasticsearch-perl/blob/5659396a3c6fcad01a85ecf92ff23804cf6eefe1/lib/Search/Elasticsearch/Client/7_0/Bulk.pm#L39-L50

You push something to the buffer and flush if it's greater than the limit. But shouldn't the limit be the limit, meaning: Not more than that?

Without testing I think, a correct way would be:

    while (@_) {
        my @json = $self->_encode_action( splice( @_, 0, 2 ) );

        my $size = $self->_buffer_size;
        my $added_size = 0;
        $added_size += length($_) + 1 for @json;
        
        $self->flush
            if ( $max_size and $size+$added_size > $max_size )  # Would the new size exceed the max_size
            || ( $max_count and $self->_buffer_count >= $max_count ) # is the current count equal or higher than the max_count?
            || ( $max_time  and time >= $self->_last_flush + $max_time );

        push @$buffer, @json;

        my $size = $added_size + $self->_buffer_size;
        $self->_buffer_count( $self->_buffer_count + 1 );
        $self->_buffer_size($size);
    }
    return 1;

Skeeve avatar Nov 19 '21 07:11 Skeeve