django-pandas
django-pandas copied to clipboard
Add support for postgres COPY TO/FROM command
It would be nice for bigger datasets to be able to utilize postgres COPY TO/FROM to speed up data IO dramatically. Here is another object manager that is set up to support it: https://github.com/palewire/django-postgres-copy
Since COPY TO/FROM reads and dumps in CSV format, all django-pandas would need to do is convert between CSV and a DataFrame.
For context, I just ran a comparison between the two operations.
Database row retrieval count: 236,170,980
to_dataframe() time: 774 seconds
COPY TO to_csv() time: 223 seconds
converting csv to dataframe time: 24 seconds