datapusher-plus
datapusher-plus copied to clipboard
Upgrade to psycopg3 and use async COPY
Psycopg3 has been stable for ~1.5 years (https://www.psycopg.org/articles/2021/10/13/psycopg-30-released/) and one of its headline features is async support.
With it, perhaps we can do async COPY so we can return quickly even for very large files, while the streaming COPY and auto-indexing is still in progress.
This should allow us to support a more predictable Resource first upload workflow, allowing us to create the placeholder resource with inferred metadata in a few seconds (<5 seconds, even for very large files).
cc @wardi @twdbben