datawhores
Ability to scrape directly to a s3 bucket.
Is your feature request related to a problem? Please describe. The ability to have files directly saved to an s3 object store as they are scraped. Including profiles, and metadata.
Describe the solution you'd like I would like to see a solution where I can spin up a docker container. Then with a few configuration options, have it pull the profile and login data from an s3 bucket, scrape content and then store that content in another s3 bucket.
Describe alternatives you've considered Currently, pushing the content via custom scripts to s3 with rclone is working but cumbersome.
Additional context Being able to scrape with a high bandwidth connection from a vps means content can be secured from the source quickly. Many VPS providers have tons of bandwidth available to their plans, yet very limited disk space. Being able to place the scraped data onto an s3 bucket as it is scraped would be ideal for my use case.
