meilisync
meilisync copied to clipboard
Postgres `get_full_data` is slow for large datasets
I found that get_full_data was unusably slow for datasets >1m rows.
I believe the culprit is the use of offset/limit pagination, which is not performant for large offsets.
I think a better implementation would be to use a server-side cursor and use fetchmany to fetch size number of rows at a time.
See Psycopg docs for more.
OK, could you please make a PR?