hatch publish reports SSL: CERTIFICATE_VERIFY_FAILED: unable to get local issuer certificate

[fleetingbytes]

Thanks to the newly implemented possibility of having repository-specific publishing options since hatch-v1.5.0, I have set multiple publishing repositories in my config.toml, similar to this:

[publish.index.repos.test]
url = "https://test.pypi.org/legacy/"
user = "__token__"
auth = "pypi-<SECRET>"
ca-cert = "/path/to/ca-bundle.crt"

[publish.index.repos.company]
url = "https://gitlab.company.local/api/v4/projects/<PROJECT-ID>/packages/pypi"
user = "gitlab+deploy-token-451"
auth = "<SECRET>"
ca-cert = "/path/to/ca-bundle.crt"

The credentials and ca-cert are the same as I have set in my .pypirc. I can successfully run hatch publish --repo company to publish the artifacts in my custom package registry, but when I try to publish it to the PyPI test repo with --repo test

hatch publish --repo test
dist\<wheel-name>.whl ... failed
[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)

However, everything works when I'm publishing the artifacts manually via python -m twine upload --repository test dist/*. Can you help me debug this? Which certificate verification fails here exactly? Is there a way to get a more verbose output when running the publish procedure?

I should add that the error happens on my company's computer which uses a proxy server. I tested the hatch publish procedure on my private computer, and from there I can successfully hatch publish to the test.pypi.org using a similar configuration (without the ca-cert option). On my company's computer the ca-cert option is necessary lest I get another error ([SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain).

[ofek]

On my company's computer the ca-cert option is necessary lest I get another error

Necessary even for test.pypi.org?

Is there a way to get a more verbose output when running the publish procedure?

There isn't much else, here's the code: https://github.com/pypa/hatch/blob/hatch-v1.5.0/src/hatch/index/core.py#L33-L76

[fleetingbytes]

Yes, my company's configuration needs the ca-cert for both upload.pypi and test.pypi. I will try to enrich the hatch source by some logging configuration and log events, in the hope for hunting this down. But I will take some time. First I'm on vacation till the rest of the week and in the upcoming month I need to work a lot of overtime, so the progress will be rather slow on this.

[ofek]

I think I'll need help debugging this

[SurenNihalani]

If you can still repro this, can you share your env? It's usually that REQUESTS_CA_BUNDLE/SSL_CERT_FILE environment variables are pointing to invalid locations or you need to install certifi

[fleetingbytes]

Sorry for the long delay, but I did not have time to debug this, until today. I've solved it now (after so many hours 😭 !).

My issue was not caused by hatch. In this stackoverflow answer I have found the instruction to downlad the Certificate chain manually from my browser and append it to the .pem file I'm using. So I did exactly that.

I downloaded pypi.org's certificate chain, appended it to the ca-cert file I point to in my hatch's config.toml ca-cert = "/path/to/ca-bundle.crt", and now I can finally publish there using hatch publish --repo test

Thank God!

[fleetingbytes]

While I might have solved the problem, I don't understand the solution. When I publish the build with twine, twine is completely happy communicating with test.pypi.org using my usual ca-cert bundle. hatch publish on the other hand, is throwing the SSL error, unless I append test.pypi's certificate chain to my ca-cert bundle. 🤔

[ofek]

Hatch uses httpx (cc @tomchristie) while Twine uses requests (cc @sethmlarson) ... any idea what might be different about test.pypi.org on some machines even though both libraries use certifi?

[ofek]

I'm glad it's fixed for you, I'd ask https://github.com/encode/httpx

[fleetingbytes]

I found the error why I could not publish to upload.pypi.org. I wrongly assumed that it would have the same certificates as test.pypi.org. Turns they have different ones and I need to have both of them in my CA-bundle. Now I can publish there with both hatch and twine.