Allow root CA bundle to be configured

[jobec]

• [ ] I have searched the issues of this repo and believe that this is not a duplicate. This is a duplicate of closed issue #790
• [x] I have searched the documentation and believe that my question is not covered.

Feature Request

Allow and alternate root CA bundle to be specified from CLI and in the global/project settings file. Similar to how pip does it. This is crucial in environments where direct access to pypi is blocked and when using internal pypi mirrors with certificates signed by an internal root CA. Or with SSL traffic inspecting firewalls.

https://pip.pypa.io/en/stable/reference/pip/?highlight=proxy#cmdoption-cert

Having to set an environment variable like REQUESTS_CA_BUNDLE for an underlying package is not user-friendly and expects the user to know what poetry does under the hood.

[MichalMazurek]

Hi, It would be also nice to disable certificate checking, I'm having a lot of issues with a self-signed certificate for private PyPI repo. I can fix that, but first, it would be great to know if this will be welcomed into the project. This can be done together with adding support for the request above.

[floer32]

can you confirm/deny whether REQUESTS_CA_BUNDLE actually works?

Having to set an environment variable like REQUESTS_CA_BUNDLE for an underlying package is not user-friendly and expects the user to know what poetry does under the hood.

Arguably that's just a documentation problem.

While encapsulation can be good, requests is extremely common, so much so that most people treat it as if it's basically part of the Python stdlib. REQUESTS_CA_BUNDLE is barely more 'under the hood' of a parameter, than PYTHONUNBUFFERED or other python-official features. So really as long as it is documented and guaranteed that requests is used, that could be enough for immediate future.

---

The bigger question, I think, is whether/how-much to expose Pip directly through Poetry.

i.e. In pytest there is the env var, PYTEST_ADDOPTS and it appends flags to any pytest invocation. there could be POETRY_PIP_ADDOPTS etc. maybe. Or instead of an env var it could be something in config; with pytest as example again, it has pytest_addopts = ... supported from its config file (.ini in that case, similar to .toml). https://docs.pytest.org/en/latest/customize.html#adding-default-options Of course it won't be exactly like pytest, it's just a worthwhile example to consider.

this under-the-hood-config question is kinda relevant to #558 (though there it is about pip wheel and --find-links configs, rather than something to do with requests or CA bundles)

---

maybe this relates to what #697 is getting at too... just generally how much to connect Poetry's high-level view to the lower-level nuts and bolts of the tools involved. vs. how much to have known-good ways to "break out" of the typical paths to customize for your needs... while remaining compatible with Poetry

[jobec]

Yes, using REQUESTS_CA_BUNDLE works.

I don’t think it’s a documentation issue though. Having to set an environment variable while there is the “poetry config” command is counter intuitive.

[drunkwcodes]

Maybe adding requests library in system requirements will make sense.

Rather than include requests config in poetry configuration, having a script to set env vars in the project sounds better.

[jobec]

From a UX point of view this isn't true. People use poetry. They shouldn't care about what's underneath.

A lot of tools use this approach (configuration options for the tool, not it's libraries) Npm, pip, git, vagrant, ... and even requests passes some of it's config on to urllib.

[drunkwcodes]

@jobec You are probably right. It should be handled by poetry.

What I considered about is the inconsistency between the env vars and the poetry CA config. The setting is used by custom certificate users but the majority poetry users only query PyPI. The setting will be often overlooked and outdated, more of a concern rather than a degree of freedom.

Placing it in env vars is fine. And we can choose handling it by python or the shell. Temporary or not. https://stackoverflow.com/questions/47285018/how-to-set-environment-variables-using-python-in-windows-linux

[kmray]

I just ran into this issue and the solution of setting REQUESTS_CA_BUNDLE will be extremely inconvenient.

We are currently using two private repositories: an internal one with a corporate certificate and a SaaS hosted one with a public certificate. If I set the REQUESTS_CA_BUNDLE to the corporate certificate, the SaaS won't work and if I set it to the regular bundle, the private one won't work.

A workaround would be to create a hybrid bundle with both in it and I could manage this on the builds but this would also require that all users set the variable to the hybrid bundle when working on the repo. Having a "cert = " field in source would solve this problem on a per source basis.

[drunkwcodes]

@kmray Is the project public to outer space?

There is no need to add field if hybrid bundle is usable to all project members.

[Caligatio]

I'm forgetting if GitHub emails people on cross ticket mentions but I have a PR, #1325, that addresses this request. The developers said in Discord that they're ranking PRs by number of votes so please +1 the PR :)

[stale[bot]]

Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[KyleKing]

:thinking: With #1325, can we use Poetry to set the CA Certificate path rather than the hackish REQUESTS_CA_BUNDLE variable? I've tried to read through the documentation/tickets and tried a number of options, but still run into the self-signed certificate error

(I'm on Windows in a corporate environment that uses a self-signed certificate)

[Caligatio]

Is this for installing packages? If so, poetry config certificates.foo.cert /path/to/ca.pem should work as long as you previously configured "foo" in your pyproject.toml using something like

[[tool.poetry.source]]
name = "foo"
url = "https://foo.bar/simple/"

There is still an outstanding bug if you have periods in your repo shortname.

[KyleKing]

Is there a user error on my part or is the feature not intended for this use case?

---

This works:

SET REQUESTS_CA_BUNDLE=C:\Users\king.kyle\certificate.pem
poetry add cerberus

---

But I'm running errors when trying:

poetry config certificates.pypi.cert C:\Users\king.kyle\certificate.pem
poetry add cerberus -vvv

(Note: also tried adding poetry config certificates.pypi.client-cert C:\Users\king.kyle\certificate.pem)

---

I added the below snippet to the toml file:

[[tool.poetry.source]]
name = "foo"
url = "https://pypi.org/"

Then tried:

poetry config certificates.foo.cert C:\Users\king.kyle\certificate.pem
poetry add cerberus -vvv

Full error:

(py375) C:\Users\king.kyle\Developer\Project>poetry add cerberus -vvv
Using virtualenv: C:\Users\king.kyle\AppData\Local\Continuum\anaconda2\envs\py375

[SSLError]
HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /pypi/cerberus/json (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1076)')))

Traceback (most recent call last):
  File "C:\Users\king.kyle\.poetry\lib\poetry\_vendor\py3.7\clikit\console_application.py", line 131, in run
    status_code = command.handle(parsed_args, io)
  File "C:\Users\king.kyle\.poetry\lib\poetry\_vendor\py3.7\clikit\api\command\command.py", line 120, in handle
    status_code = self._do_handle(args, io)
  File "C:\Users\king.kyle\.poetry\lib\poetry\_vendor\py3.7\clikit\api\command\command.py", line 171, in _do_handle
    return getattr(handler, handler_method)(args, io, self)
  File "C:\Users\king.kyle\.poetry\lib\poetry\_vendor\py3.7\cleo\commands\command.py", line 92, in wrap_handle
    return self.handle()
  File "C:\Users\king.kyle\.poetry\lib\poetry\console\commands\add.py", line 89, in handle
    packages, allow_prereleases=self.option('allow-prereleases')
  File "C:\Users\king.kyle\.poetry\lib\poetry\console\commands\init.py", line 303, in _determine_requirements
    requirement['name'], allow_prereleases=allow_prereleases
  File "C:\Users\king.kyle\.poetry\lib\poetry\console\commands\init.py", line 333, in _find_best_version_for_package
    name, required_version, allow_prereleases=allow_prereleases
  File "C:\Users\king.kyle\.poetry\lib\poetry\version\version_selector.py", line 29, in find_best_candidate
    package_name, constraint, allow_prereleases=True
  File "C:\Users\king.kyle\.poetry\lib\poetry\repositories\pool.py", line 149, in find_packages
    name, constraint, extras=extras, allow_prereleases=allow_prereleases
  File "C:\Users\king.kyle\.poetry\lib\poetry\repositories\pypi_repository.py", line 112, in find_packages
    info = self.get_package_info(name)
  File "C:\Users\king.kyle\.poetry\lib\poetry\repositories\pypi_repository.py", line 265, in get_package_info
    name, lambda: self._get_package_info(name)
  File "C:\Users\king.kyle\.poetry\lib\poetry\_vendor\py3.7\cachy\repository.py", line 174, in remember_forever
    val = value(callback)
  File "C:\Users\king.kyle\.poetry\lib\poetry\_vendor\py3.7\cachy\helpers.py", line 6, in value
    return val()
  File "C:\Users\king.kyle\.poetry\lib\poetry\repositories\pypi_repository.py", line 265, in <lambda>
    name, lambda: self._get_package_info(name)
  File "C:\Users\king.kyle\.poetry\lib\poetry\repositories\pypi_repository.py", line 269, in _get_package_info
    data = self._get('pypi/{}/json'.format(name))
  File "C:\Users\king.kyle\.poetry\lib\poetry\repositories\pypi_repository.py", line 364, in _get
    json_response = self._session.get(self._url + endpoint)
  File "C:\Users\king.kyle\.poetry\lib\poetry\_vendor\py3.7\requests\sessions.py", line 546, in get
    return self.request('GET', url, **kwargs)
  File "C:\Users\king.kyle\.poetry\lib\poetry\_vendor\py3.7\requests\sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Users\king.kyle\.poetry\lib\poetry\_vendor\py3.7\requests\sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "C:\Users\king.kyle\.poetry\lib\poetry\_vendor\py3.7\cachecontrol\adapter.py", line 53, in send
    resp = super(CacheControlAdapter, self).send(request, **kw)
  File "C:\Users\king.kyle\.poetry\lib\poetry\_vendor\py3.7\requests\adapters.py", line 514, in send
    raise SSLError(e, request=request)


(py375) C:\Users\king.kyle\Developer\Project>

[Caligatio]

If you have

[[tool.poetry.source]]
name = "foo"
url = "https://pypi.org/"

then you must then do poetry config certificates.foo.cert C:\Users\king.kyle\certificate.pem (note the "foo" part). The "foo" is the part that relates a CA to a given custom repo.

[KyleKing]

Yeah, that was what I tried. My last comment could have been ordered in reverse since the third example was this approach.

(py375) C:\Users\king.kyle\Developer\Projectpoetry config --list
certificates.foo.cert = "C:\\Users\\king.kyle\\certificate.pem"  # None
virtualenvs.create = true
virtualenvs.in-project = false
virtualenvs.path = "{cache-dir}\\virtualenvs"  # C:\Users\king.kyle\AppData\Local\pypoetry\Cache\virtualenvs

(py375) C:\Users\king.kyle\Developer\Project

[Caligatio]

Apologies for missing that last bit. I think I have a good idea what might be happening; can you try (note the new default line and the non-pypi name):

[[tool.poetry.source]]
name = "foo"
url = "https://pypi.org/"
default = true

Then do poetry config certificates.foo.cert C:\Users\king.kyle\certificate.pem

PyPi is special amongst repository peers in Poetry in that it bypasses my custom certificate logic. It also always gets consulted to see if a package is available. I believe it's probably doing the "foo" lookup correctly but, since you're squatting on the PyPi domain, the special PyPi lookup is failing. The default = true line should disable the default PyPi repo and only use your custom "foo" repo.

[KyleKing]

Thanks, making progress!

I created a new project poetry new TestPyPiSelfSigned, added [[tool.poetry.source]] with default = true to the pyproject.toml, and set certificates.foo.cert

Those steps appear to resolve the self-signed errors, but poetry appears unable to connect to pypi to identify package versions. Unseting the poetry configuration causes the self-signed errors to reappear

(py373) C:\Users\king.kyle\Developer\TestPyPiSelfSigned>poetry install -vvv
Using virtualenv: C:\Users\king.kyle\AppData\Local\Continuum\anaconda2\envs\py373
Updating dependencies
Resolving dependencies...
   1: fact: testpypiselfsigned is 0.1.0
   1: derived: testpypiselfsigned
   1: fact: testpypiselfsigned depends on pytest (^5.2)
   1: fact: testpypiselfsigned depends on pytest (^5.2)
   1: selecting testpypiselfsigned (0.1.0)
   1: derived: pytest (^5.2)
   1: fact: no versions of pytest match >=5.2,<6.0
   1: conflict: no versions of pytest match >=5.2,<6.0
   1: ! pytest (^5.2) is satisfied by pytest (^5.2)
   1: ! which is caused by "testpypiselfsigned depends on pytest (^5.2)"
   1: ! thus: version solving failed
   1: Version solving took 0.328 seconds.
   1: Tried 1 solutions.

[SolverProblemError]
Because testpypiselfsigned depends on pytest (^5.2) which doesn't match any versions, version solving failed.

Traceback (most recent call last):
  File "C:\Users\king.kyle\.poetry\lib\poetry\_vendor\py3.7\clikit\console_application.py", line 131, in run
    status_code = command.handle(parsed_args, io)
  File "C:\Users\king.kyle\.poetry\lib\poetry\_vendor\py3.7\clikit\api\command\command.py", line 120, in handle
    status_code = self._do_handle(args, io)
  File "C:\Users\king.kyle\.poetry\lib\poetry\_vendor\py3.7\clikit\api\command\command.py", line 171, in _do_handle
    return getattr(handler, handler_method)(args, io, self)
  File "C:\Users\king.kyle\.poetry\lib\poetry\_vendor\py3.7\cleo\commands\command.py", line 92, in wrap_handle
    return self.handle()
  File "C:\Users\king.kyle\.poetry\lib\poetry\console\commands\install.py", line 63, in handle
    return_code = installer.run()
  File "C:\Users\king.kyle\.poetry\lib\poetry\installation\installer.py", line 74, in run
    self._do_install(local_repo)
  File "C:\Users\king.kyle\.poetry\lib\poetry\installation\installer.py", line 161, in _do_install
    ops = solver.solve(use_latest=self._whitelist)
  File "C:\Users\king.kyle\.poetry\lib\poetry\puzzle\solver.py", line 36, in solve
    packages, depths = self._solve(use_latest=use_latest)
  File "C:\Users\king.kyle\.poetry\lib\poetry\puzzle\solver.py", line 190, in _solve
    raise SolverProblemError(e)

(py373) C:\Users\king.kyle\Developer\TestPyPiSelfSigned>poetry config certificates.foo.cert --unset

(py373) C:\Users\king.kyle\Developer\TestPyPiSelfSigned>poetry install
Updating dependencies
Resolving dependencies...

[SSLError]
HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /pytest/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1056)')))

[KyleKing]

Ooh! Actually solved it, the path should be https://pypi.org/project/

[[tool.poetry.source]]
name = "pypi_with_cert"
url = "https://pypi.org/project/"
default = true

poetry config certificates.pypi_with_cert.cert C:\Users\king.kyle\certificate.pem

Would it be useful to add this to the documentation somewhere?

Edit: see comment below (https://github.com/python-poetry/poetry/issues/1012#issuecomment-567061688), the url should be url = "https://pypi.org/simple/"

[KyleKing]

Hmm, I'm running into an error with those changes. Seems to work fine if the version specification matches the latest, but poetry can't identify prior versions. For example, changing the pytest = "^5.0" to pytest = "^4.0" causes the install to fail. Is this an issue with the url ("https://pypi.org/project/")?

Create new project

poetry new TestPyPiSelfSigned

Modify the toml file

[tool.poetry]
name = "TestPyPiSelfSigned"
version = "0.1.0"
description = ""
authors = ["Kyle King"]

[[tool.poetry.source]]
name = "pypi_with_cert"
url = "https://pypi.org/project/"
default = true

[tool.poetry.dependencies]
python = "^3.7"

[tool.poetry.dev-dependencies]
pytest = "^4.0"

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

Try to install or update

(py373) C:\Users\king.kyle\Developer\TestPyPiSelfSigned>poetry update -vvv
Using virtualenv: C:\Users\king.kyle\AppData\Local\Continuum\anaconda2\envs\py373
Updating dependencies
Resolving dependencies...
   1: fact: testpypiselfsigned is 0.1.0
   1: derived: testpypiselfsigned
   1: fact: testpypiselfsigned depends on pytest (^4.0)
   1: fact: testpypiselfsigned depends on pytest (^4.0)
   1: selecting testpypiselfsigned (0.1.0)
   1: derived: pytest (^4.0)
pypi_with_cert: 0 packages found for pytest >=4.0,<5.0
   1: fact: no versions of pytest match >=4.0,<5.0
   1: conflict: no versions of pytest match >=4.0,<5.0
   1: ! pytest (^4.0) is satisfied by pytest (^4.0)
   1: ! which is caused by "testpypiselfsigned depends on pytest (^4.0)"
   1: ! thus: version solving failed
   1: Version solving took 0.491 seconds.
   1: Tried 1 solutions.

[SolverProblemError]
Because testpypiselfsigned depends on pytest (^4.0) which doesn't match any versions, version solving failed.

Traceback (most recent call last):
  File "C:\Users\king.kyle\.poetry\lib\poetry\_vendor\py3.7\clikit\console_application.py", line 131, in run
    status_code = command.handle(parsed_args, io)
  File "C:\Users\king.kyle\.poetry\lib\poetry\_vendor\py3.7\clikit\api\command\command.py", line 120, in handle
    status_code = self._do_handle(args, io)
  File "C:\Users\king.kyle\.poetry\lib\poetry\_vendor\py3.7\clikit\api\command\command.py", line 171, in _do_handle
    return getattr(handler, handler_method)(args, io, self)
  File "C:\Users\king.kyle\.poetry\lib\poetry\_vendor\py3.7\cleo\commands\command.py", line 92, in wrap_handle
    return self.handle()
  File "C:\Users\king.kyle\.poetry\lib\poetry\console\commands\update.py", line 49, in handle
    return installer.run()
  File "C:\Users\king.kyle\.poetry\lib\poetry\installation\installer.py", line 74, in run
    self._do_install(local_repo)
  File "C:\Users\king.kyle\.poetry\lib\poetry\installation\installer.py", line 161, in _do_install
    ops = solver.solve(use_latest=self._whitelist)
  File "C:\Users\king.kyle\.poetry\lib\poetry\puzzle\solver.py", line 36, in solve
    packages, depths = self._solve(use_latest=use_latest)
  File "C:\Users\king.kyle\.poetry\lib\poetry\puzzle\solver.py", line 190, in _solve
    raise SolverProblemError(e)


(py373) C:\Users\king.kyle\Developer\TestPyPiSelfSigned>

[KyleKing]

Ok, final answer? Sorry for the spam. I believe we want the URL "https://pypi.org/simple/" (based on documentation from the Link class)

https://github.com/python-poetry/poetry/blob/6b09639002e737e76a09eb973f0049262511d3f0/poetry/packages/utils/link.py#L17

This now works

[[tool.poetry.source]]
name = "pypi_with_cert"
url = "https://pypi.org/simple/"
default = true

Then run: poetry config certificates.pypi_with_cert.cert C:\Users\Path\To\certificate.pem

[Caligatio]

Huzzah! As a rule of thumb, you'll want /simple for most repositories.

@sdispater: Do you have any strong feelings one way or the other about allowing a certificate authority override for PyPI? I'm happy to do a PR with the relevant changes if you're supportive.

[sdispater]

@Caligatio What is the use case for overriding the certificate authority for PyPI?

[jobec]

Being behind a corporate firewall that does TLS interception

[Caligatio]

Either what @jobec said or if you're in a corporate environment that squats on external domain names.

My first thought on this is that it seems like a not-great security idea but, on the other hand, it would require explicit settings to enable and thus shouldn't be an accident. We got something working for @KyleKing in this thread but it wasn't exactly obvious what the actual problem was.

[jobec]

TLS interception is usually done as a security measure. It allows to virus/threath scan what’s being downloaded. Whether it’s effective is a different discussion, but in many companies it’s a policy.

And I agree, this can’t be configured by accident and you need to know how and where to get such root CA bundle. So the chance of abuse is low.

[KyleKing]

Unfortunately, I'm not privy to what changed in the IT Infrastructure. There was a major overhaul in the last two years that caused us to start seeing the error "Self-Signed Certificate" whenever attempting to make a request with pip

Creating a pem file isn't difficult, but you need to know the steps, so probably at least 60% of the people using this workaround would be my coworkers

[jerodg]

I couldn't get this to work on mac through the proxy. Pip is configured with our internal certificate and works fine but I can't even get poetry to install due to the self-signed certificate error.

Are there any workarounds to install through proxy?

[Caligatio]

@jerodg Are you having troubles installing poetry or installing packages using Poetry? If the prior, how you installing it? I typically use pip to bootstrap Poetry and that sounds like it works for you.

[jerodg]

@Caligatio I can't even install it because of the proxy. I know this thread was mostly about packages but I didn't see any other issues relating to it. It throws a urllib error.

[jobec]

This doesn't work as expected it seems.

poetry config certificates.private_pypi.cert /project/certificates/ca-bundle.pem --local

It stores the certificate config in /home/user/.config/pypoetry/auth.toml instead of local to the project without giving warnings or anything.

Manually copying the config from that file to the poetry.toml file in the project root works though. Allowing you to configure it at the project level.