Allow setting open build env vars in `.readthedocs.yml` config file

[webknjaz]

Details

• Read the Docs project URL: N/A
• Build URL (if applicable): N/A
• Read the Docs username (if applicable): N/A

Expected Result

I want to be able to set public env vars via .readthedocs.yml.

Actual Result

I can only do this in the web UI.

Use-case

I have an in-tree Sphinx extension and import paths are set up via tox. Since RTD doesn't allow specifying a custom build tool and has a hardcoded command, I had to seek another way of making it importable. I've chosen to set a PYTHONPATH variable to address this. But I'd prefer an explicit way of doing it.

Bonus thoughts

Setting "special" env vars may conflict with RTD's special values. It may be a good idea to enable first-class support for adding custom import location as a YAML sequence in the config and then properly merging it with the existing PYTHONPATH.

[flying-sheep]

Other use case: installing packages from custom public indices via PIP_FIND_LINKS="https://wheels.example.com" where e.g.

$ curl $PIP_FIND_LINKS
<a href="my_special_use_pkg-0.4-py3-none-any.whl">my-special-use-pkg</a>

[agjohnson]

Though we have this functionality in our UI, this seems like a great feature for our config file. This is handy for non-secret environment variables, as noted, but also for helping increase the odds of a build being reproducible without extra project configuration.

I raised issues with the scope of environment variables defined in our configuration through build.jobs. @humitos would you agree this feature makes sense without being at odds with the plans for build.jobs?

[humitos]

@agjohnson

I raised issues with the scope of environment variables defined in our configuration through build.jobs. @humitos would you agree this feature makes sense without being at odds with the plans for build.jobs?

Yes. I think this feature makes sense and can work well together with build.jobs. All the environment variables will be loaded after parsing the config file and will be passed to all commands (default ones ran by Read the Docs and custom ones defined by users)

[agjohnson]

Great! This seems like a great feature to add, I'm putting this on our 2022Q2 roadmap :+1:

[agjohnson]

I lied, we didn't get to this in 2022. I'm going to keep this on the backlog for now.

[ericholscher]

Also just a note, I think you can probably hack this with something like:

build
  jobs:
    pre_build: echo "export FOO=bar" >> ~/.bash_profile

No guarantee it continues to work though 🙃

[humitos]

@ericholscher I tested your idea, but it seems it doesn't work as expected 😞 . I quickly tested this at https://readthedocs.org/projects/test-builds/builds/18714592/

I suppose it's because each command is executed with /bin/sh and it not loads any of the bash initial files: https://github.com/readthedocs/readthedocs.org/blob/fba14285d28913aac29a640e5e2761d76c7e8145/readthedocs/doc_builder/environments.py#L374-L379

[agjohnson]

sh only evaluates .profile and friends if it's a login shell (sh -l/sh --login), so this wouldn't work without some more creative solutions

[humitos]

@agjohnson would it be safe to pass --login? If the answer is true, that this is all we need, I'd 👍🏼 on moving forward with this. At least, it will give users a workaround for this use case.

[agjohnson]

That is a really good question! I don't know for certain here, it would be worth a test.

Executing in a login shell would open the door to using direnv, which I might suggest over a solution like :point_up:

It's also possible to author a local .profile in the repo and copy that into the /home/docs path, which is probably okay too.

[humitos]

This is issue is kind of related to https://github.com/readthedocs/readthedocs.org/issues/10462. We should consider that case when working on this idea.

[pllim]

FWIW, I was just looking for this feature. I was trying to set NUMPY_EXPERIMENTAL_DTYPE_API for some weird package I am trying to build for (https://github.com/spacetelescope/spherical_geometry/pull/241).

[humitos]

@pllim you can define the environment variables for the project in the meanwhile. Read more at https://docs.readthedocs.io/en/stable/environment-variables.html

[RemDelaporteMathurin]

Hi @humitos I recently bumped into this issue, ideally I'd like to configure everything from the config file and not have to fiddle with the UI, for reproducibility purposes. Has there been any progress on this front?

[humitos]

I'd like to configure everything from the config file and not have to fiddle with the UI

We have discussed this multiple times and there are a few reasons why we won't allow this. Mainly, versions configs vs. project configs. I don't have the issue at hand now, but if you search in the repository you will find it --in case you want to have the full context.

[agjohnson]

That feels like a bit of a separate conversation though. We can accomplish configuration file environment variables without a project level configuration file. Project level env variables could be exported first, and then version configuration file env vars exported after -- allowing version env vars to override project env vars.

Environment variables in our configuration file would be much more limited in scope though: they won't be encrypted and they have to be added to ever version's configuration file. So not useful for adding a GitHub token, but still useful for defining environment variables that alter the build process -- like the PYTHONPATH reported originally.

I don't see us reimplementing environment variables in the configuration file to help avoid using the UI, but do see value in allowing variables to come from the configuration file.

This seems like a fine compromise and would still be a useful addition to the configuration file, with the only affordance we really need to make is a configuration schema change and a build process addition.