Unable to build documentation in proxy-only network environments using nox

[brlin-tw]

I am in a network environment where internet access is only possible via a forward proxy service, and I noticed that the documentation build will error in the following occasions including but not limited to:

Session checkers(docs-build):

../../hacking/build-ansible.py docs-build core -o rst                                                                    
--> Standard Error                                                                                                       
[WARNING]: You are running the development version of Ansible. You should only                                           
run Ansible from "devel" if you are modifying the Ansible engine, or trying out                                          
features under development. This is a rapidly changing source of code and can
become unstable at any point.
/home/brlin/Workarounds/bindfs/ansible-documentation/.nox/checkers-docs-build/lib/python3.12/site-packages/antsibull_core
/utils/http.py:85: UserWarning: aio_session.get('https://galaxy.ansible.com/api/', headers={'Accept': 'application/json'}
) failed with status code Cannot connect to host galaxy.ansible.com:443 ssl:default [Temporary failure in name resolution
], retrying...
  warnings.warn(
/home/brlin/Workarounds/bindfs/ansible-documentation/.nox/checkers-docs-build/lib/python3.12/site-packages/antsibull_core
/utils/http.py:85: UserWarning: aio_session.get('https://galaxy.ansible.com/api/', headers={'Accept': 'application/json'}
) failed with status code Cannot connect to host galaxy.ansible.com:443 ssl:default [Temporary failure in name resolution
], finally failed.
  warnings.warn(
Traceback (most recent call last):
  File "/tmp/docs-build-7s0cud__-sanity/docs/docsite/../../hacking/build-ansible.py", line 130, in <module>
    main()

../../hacking/build-ansible.py docs-build core -o rst:

../../hacking/build-ansible.py docs-build core -o rst                                                                    
/home/brlin/Workarounds/bindfs/ansible-documentation/.nox/make/lib/python3.12/site-packages/antsibull_core/utils/http.py:
85: UserWarning: aio_session.get('https://galaxy.ansible.com/api/', headers={'Accept': 'application/json'}) failed with s
tatus code Cannot connect to host galaxy.ansible.com:443 ssl:default [Temporary failure in name resolution], retrying...

I have set and exported the following proxy-related environment variables:

$ printenv | grep -i proxy
no_proxy=localhost
https_proxy=http://192.168.49.1:8228
NO_PROXY=localhost
HTTPS_PROXY=http://192.168.49.1:8228
HTTP_PROXY=http://192.168.49.1:8228
http_proxy=http://192.168.49.1:8228

and have the Docker daemon configured to use the proxy service:

{
  "proxies": {
    "http-proxy": "http://192.168.49.1:8228",
    "https-proxy": "http://192.168.49.1:8228",
    "no-proxy": "localhost,127.0.0.0/8"
  }
}

It would be great if these settings are fully honored so that the build can be completed without switching network environments. Thanks in advance!

[oraNod]

@felixfontein FYI, this one might be of interest to you as there are antsibull_core messages in logs. Although they might be coming from underlying packages.

[felixfontein]

antsibull-core and antsibull-docs use aiohttp (https://pypi.org/project/aiohttp/) for network requests. It seems to be possible to configure proxy usage there: https://docs.aiohttp.org/en/stable/client_advanced.html#proxy-support I'm not sure if there's support for environment variables like the ones mentioned above, I guess not (otherwise it should have worked I guess).

[brlin-tw]

@felixfontein

I'm not sure if there's support for environment variables like the ones mentioned above, I guess not (otherwise it should have worked I guess).

It seems to be supported ~~by~~ but not enabled by default:

[felixfontein]

trust_env=True sounds like something we could enable. @gotmax23 what do you think?

Edit: we could also make this configurable (with default True, or to be more conservative, with default False; I think True would be ok).

[gotmax23]

trust_env=True sounds like something we could enable. @gotmax23 what do you think?

Yeah, I think it makes sense to have that as the default.

Edit: we could also make this configurable (with default True, or to be more conservative, with default False; I think True would be ok).

Unless there's a strong reason to add a config that I'm missing, I'd prefer to just keep this simple.

[x1101]

Coming from the Systems side of things, I'd expect applications to implicitly respect my proxy settings. So, defaulting to True for this seems like a perfectly reasonable solution. If we find any use-cases where this is problematic, we can always come back and add a configuration option (but leave it defaulted to True)

[felixfontein]

antsibull-docs 2.15.0 now supports env variables for proxy configuration. #1994 updates the dependencies for this repo to use this version.

[brlin-tw]

I have checked out the new code and can verify that the fix works, thanks!

[felixfontein]

Great to hear! :)