vulnerablecode icon indicating copy to clipboard operation
vulnerablecode copied to clipboard

Run importers in Parallel

Open ziadhany opened this issue 2 years ago • 1 comments

Instead of running ./manage.py import --all in sequence We can use multiprocessing and try to use all CPU power.

suggestions:

  • we can use Celery and schedule tasks and run easily importers in parallel (another benefit: we can run periodic Tasks and make importers synchronize )

  • we can make it simple and use some code from scancode.

this could be a solution for #833

ziadhany avatar Aug 09 '22 19:08 ziadhany

While I was trying to run importers using threads I couldn't notes any improvement ! https://stackoverflow.com/questions/31062124/multiple-db-inserts-with-django-performance-is-not-increased-by-parallel-threads

ziadhany avatar Aug 15 '22 13:08 ziadhany