scrapyd-client
scrapyd-client copied to clipboard
scrapy
Command line client for Scrapyd server
Most CD pipeline frameworks heavily use environnent variables to store secrets and configuration. Currently, we need to either store these secrets in `scrapy.cfg` and commit to the repo or substitute...
When using scrapyd-deploy, the command uses the authentication configuration in `scrapy.cfg` (username/password) to perform BasicAuth. When using other commands, it fails. ```sh $ scrapyd-client projects Received a malformed response: Unauthorized...
Mostly `scrapyd-deploy` options and exception handling. Presently around 82%.
`scrapyd-client -t ` `target` here needs to be a base URL. With `scrapyd-deploy`, it is instead the name of the target in the configuration file. I don't see the point...
I fail to see how I should "Eggifying" a project that uses DjangoItem. Am i supposed to bundle the django project too within the egg ?
With the addition of `scrapyd-client`, it's unusual for a flag to perform an action.
We should remove `--list-projects` in the next version.
We already have `listprojects`, `listspiders`, `schedule`, `addversion`. https://scrapyd.readthedocs.io/en/stable/api.html Idempotent: * [ ] daemonstatus * [ ] listversions * [ ] listjobs The commands would omit the `list` prefix, like existing...
### Prefase 📝 I am using scrapyd-client utility to deploy spiders on prod server. In case to avoid my spiders to be banned, IT support configured forward proxy to hide...
