dosage icon indicating copy to clipboard operation
dosage copied to clipboard

Scheduling & performance

Open TobiX opened this issue 5 years ago • 2 comments

Currently, dosage downloads comic in a very straightforward way:

  1. Get page
  2. Parse page
  3. Get images
  4. Continue with next page

For better performance, the user can decide to run download multiple comics in parallel (via the -p option) - but that's more of a clutch - the threads aren't aware of each other, which could lead to the situation where multiple threads fetch comics from the same hoster.

We should evaluate a better scheduling system, satisfying at least the following requirements:

  • [ ] Parallel downloads from multiple hosts
  • [ ] Throttling per host (we don't want to overload a hoster)
  • [ ] Image downloads can be handled separate from page parsing

It might be worthwhile to look at things like asyncio, async/await or something like that...

TobiX avatar Dec 02 '19 22:12 TobiX