vulnerablecode
vulnerablecode copied to clipboard
Run importers in Parallel
Instead of running ./manage.py import --all
in sequence
We can use multiprocessing and try to use all CPU power.
suggestions:
-
we can use Celery and schedule tasks and run easily importers in parallel (another benefit: we can run periodic Tasks and make importers synchronize )
-
we can make it simple and use some code from scancode.
this could be a solution for #833
While I was trying to run importers using threads I couldn't notes any improvement ! https://stackoverflow.com/questions/31062124/multiple-db-inserts-with-django-performance-is-not-increased-by-parallel-threads