meilisync icon indicating copy to clipboard operation
meilisync copied to clipboard

Postgres `get_full_data` is slow for large datasets

Open MattExact opened this issue 1 year ago • 1 comments

I found that get_full_data was unusably slow for datasets >1m rows. I believe the culprit is the use of offset/limit pagination, which is not performant for large offsets.

I think a better implementation would be to use a server-side cursor and use fetchmany to fetch size number of rows at a time. See Psycopg docs for more.

MattExact avatar May 20 '24 14:05 MattExact

OK, could you please make a PR?

long2ice avatar May 20 '24 14:05 long2ice