datapusher-plus icon indicating copy to clipboard operation
datapusher-plus copied to clipboard

Upgrade to psycopg3 and use async COPY

Open jqnatividad opened this issue 2 years ago • 3 comments

Psycopg3 has been stable for ~1.5 years (https://www.psycopg.org/articles/2021/10/13/psycopg-30-released/) and one of its headline features is async support.

With it, perhaps we can do async COPY so we can return quickly even for very large files, while the streaming COPY and auto-indexing is still in progress.

This should allow us to support a more predictable Resource first upload workflow, allowing us to create the placeholder resource with inferred metadata in a few seconds (<5 seconds, even for very large files).

cc @wardi @twdbben

jqnatividad avatar Feb 03 '23 12:02 jqnatividad