nautobot-app-ssot icon indicating copy to clipboard operation
nautobot-app-ssot copied to clipboard

Update Contrib to use bulk CRUD operations for Django database models

Open Renrut5 opened this issue 1 year ago • 5 comments

Environment

  • Nautobot version:
  • nautobot-ssot version:

The contrib code currently performs CRUD operations one at a time. This can take a very long time in instances where the operations get into the thousands. Part of the reason

Proposed Functionality

Take advantage of Django's bulk CRUD methods (bulk_update(), bulk_create(), etc.) in SSoT Contrib.

Use Case

Use case includes any SSoT job syncing data into Nautobot in order to improve time required to create many objects.

Renrut5 avatar Jul 15 '24 19:07 Renrut5

This is a good performance improvement idea. Bulk operations have different ways of validating / verifying during clean up - let's ensure this is not missed!

mzbroch avatar Jul 16 '24 07:07 mzbroch

Have been discussing with @jdrew82 a bit on this and we both agree the changes would need stay within the SSoT app and not impact the diffsync library.

Renrut5 avatar Jul 16 '24 13:07 Renrut5

This is something I think would be useful but would definitely have to be configurable as I wouldn't like for it to be the default behaviour.

Kircheneer avatar Jul 23 '24 09:07 Kircheneer

This is something I think would be useful but would definitely have to be configurable as I wouldn't like for it to be the default behaviour.

Absolutely should NOT be the default behavior to use bulk. That should only be used if designated by the dev.

jdrew82 avatar Jul 23 '24 14:07 jdrew82

See #555 and #34

Kircheneer avatar Dec 12 '24 13:12 Kircheneer