django-pandas icon indicating copy to clipboard operation
django-pandas copied to clipboard

Add support for postgres COPY TO/FROM command

Open DeeeeLAN opened this issue 1 year ago • 0 comments

It would be nice for bigger datasets to be able to utilize postgres COPY TO/FROM to speed up data IO dramatically. Here is another object manager that is set up to support it: https://github.com/palewire/django-postgres-copy

Since COPY TO/FROM reads and dumps in CSV format, all django-pandas would need to do is convert between CSV and a DataFrame.

For context, I just ran a comparison between the two operations.

Database row retrieval count: 236,170,980 to_dataframe() time: 774 seconds COPY TO to_csv() time: 223 seconds converting csv to dataframe time: 24 seconds

DeeeeLAN avatar Dec 05 '24 00:12 DeeeeLAN