crawlee-python icon indicating copy to clipboard operation
crawlee-python copied to clipboard

Create a new guide for scaling the crawlers

Open vdusek opened this issue 5 months ago • 5 comments

  • We could create a new documentation guide for scaling the crawlers (mainly the features from _autoscaling subpackage).
  • The guide should include the following:
    • ConcurrencySettings - how users can configure the concurrency of requests.
    • Add a short explanation of the internal components:
      • Snapshotter,
      • AutoscaledPool.
  • Inspiration: https://crawlee.dev/docs/guides/scaling-crawlers
  • How to name it? I used "request scaling" here, but it is not precise enough.

vdusek avatar Aug 30 '24 11:08 vdusek