expand environment variables in scrapy.cfg

[mxdev88]

Most CD pipeline frameworks heavily use environnent variables to store secrets and configuration. Currently, we need to either store these secrets in scrapy.cfg and commit to the repo or substitute some config values dynamically in scrapy.cfg before calling scrapyd-deploy.

If scrapyd-client was able to expand environment variables, it would make it much easier to integrate with pipelines.

Example, we could commit below scrapy.cfg file to our repository preventing issues described above.

[deploy]
url = $SCRAPYD_URL
project = my_project
username = $SCRAPYD_USERNAME
password = $SCRAPYD_PASSWORD

And scrapyd-client commands would automatically expand environment variables.

[jpmckinney]

Should we add support for this $VAR syntax to scrapy.cfg, or instead just have environment variables be another way of specifying these values (at a higher priority than the file)?

[mxdev88]

It would work as well by reading directly environment variables. How would we handle/define more complex cases with multiple targets? It would require a naming convention.

EDIT The reason why I propose to use scrapy.cfg config file is because it may be easier to implement by relying on ConfigParser interpolation with a custom one inspired from there.

[mxdev88]

hey @jpmckinney, I've PR'ed this idea in #106. Let me know what you think. thanks!

[mxdev88]

@jpmckinney do you have any more comments or suggestions on this?

[jpmckinney]

@mxdev88 Sorry, I haven't had much time for my maintainer role, but this is on my list.

[mxdev88]

hi @jpmckinney do you think you will get a chance to review at all?

[jpmckinney]

Done!