Improve dependencies parsing performance

[vladfedoriuk]

The dependencies parsing for repos is quite slow.

The reasons for this are:

• To parse dependencies for repo, a repo needs to be cloned into a temporary directory:
  • A temporary directory is created before cloning the repo and destroyed immediately after the parsing is finished
• A new process is spawned for each repo to use a tool to parse the dependencies
• Database operations - updating the repo, creating dependencies, repo-dependencies m2m relations.

There are some heuristics in place to prevent unnecessary parsing:

• A revisions hash is stored from the last repo parsing session - so if the revision does not change, we do nothing.

Possible improvements could be:

• Using a process pool to reuse the processes for dependencies parsing sessions.