fluent-plugin-sql icon indicating copy to clipboard operation
fluent-plugin-sql copied to clipboard

Published 20 hours ago • verified publisher icon fluent

All rows are read into memory prior flushing

Open static-max opened this issue 9 years ago • 6 comments

When I select a large table with no row limit, all rows a read into memory and the process never finishes. It would be nice if there was some kind of batch/cursor read method, which flushes the rows before reading all of them.

static-max avatar Dec 23 '15 15:12 static-max

If you want to read big table in batch way, using embulk or something is better.

repeatedly avatar Dec 23 '15 17:12 repeatedly

Does someone know how to use cursor in activerecord?

repeatedly avatar Jan 15 '16 03:01 repeatedly

There are some gems for SQL cursor.

  • https://github.com/afair/postgresql_cursor
  • https://github.com/zoocasa/activerecord-postgresql-cursors

okkez avatar Jan 25 '16 06:01 okkez

Activerecord-postgresql-cursors seems dead. I will try postgresql_cursor gem.

Are there no mysql cursor gem?

repeatedly avatar Feb 13 '16 08:02 repeatedly

embulk does not allow incremental importing (using state file like this plugin) and has no fluentd support as far as I can see. I forked this repo and made a working fork that uses activerecord batch to allow fetch. I make some more tests and propose a pull request.

static-max avatar Feb 15 '16 08:02 static-max

embulk does not allow incremental importing (using state file like this plugin)

Ah, I see...

uses activerecord batch

Use find_in_batches or similar method? I wait your patch.

repeatedly avatar Feb 15 '16 09:02 repeatedly