crawlee-python icon indicating copy to clipboard operation
crawlee-python copied to clipboard

fix!: Added the always-enqueue paramter to bypass deduplication

Open shivansh-bhatnagar18 opened this issue 4 months ago • 0 comments

Description

  • Earlier the requests that were generated from Request.from_url with the same unique_key generated on the same URLs were considered identical requests but this parameter if set to true allows deduplication of requests by creating a new unique_key for each request on a consistent URL

Issues

  • Closes: #547

Testing

  • The changes can be checked by creating requests for the same URL and each time this is performed, it should perform its operations without considering it as an identical request if always_enqueue is labelled true

Checklist

  • [x] CI passed

shivansh-bhatnagar18 avatar Oct 14 '24 11:10 shivansh-bhatnagar18