django-postgres-copy icon indicating copy to clipboard operation
django-postgres-copy copied to clipboard

drop indices/constraints default should be False

Open jesusch opened this issue 5 years ago • 3 comments

while using this lib I really shot our database in the head bc those are default to True we are bulk inserting small pieces (40krows) into a ~400mio table every minute dropping indices and recreating them all and all over again might not be the best default

jesusch avatar Mar 26 '20 11:03 jesusch

This is an interesting question. I'm not familiar enough with the use cases of our user base to know what people would prefer here. The use case that led me to create the library is one where the entire database is flushed with each run, so I wasn't thinking about this.

palewire avatar Dec 11 '21 12:12 palewire

My particular usecase is similar to @jesusch, especially if "upsert" is merged in. Removing and re-applying large indexes/constraints can really be a slowdown, where I'm under the impression that the intention of setting their defaults to True is to speed up things.

I think I'm +1 for "opt in" and setting defaults to False.

geoffrey-eisenbarth avatar Mar 05 '23 23:03 geoffrey-eisenbarth

@palewire In regards to PR #156 , a command like ON CONFLICT FROM CONSTRAINT "constraint_name" DO UPDATE SET ... will fail if the (model-level) constraint is removed (said PR also implements "model-level" indexes and constraints via the Meta object).

I feel like this is another sign that maybe "opt in" for drop_constraints and drop_indexes might be best.

If you feel strongly against this, then I would need to add some code in between the from_csv method (which receives the drop_constraints and drop_indexes arguments) and the instantiation of CopyMapping (which is where on_conflict is defined).

Thoughts?

geoffrey-eisenbarth avatar Mar 08 '23 17:03 geoffrey-eisenbarth