Twine 6.2.0 breaking changes: Remove hacks that support --skip-existing for indexes other than PyPI and TestPyPI.

[ppearcy]

Publishes to AWS CodeArtifact are failing with twine 6.2.0. It appears publib pulls latest twine version and then sets --skip-existing flag which results in this error now on CodeArtifact:

ERROR    UnsupportedConfiguration: The configured repository
--
'https://******.d.codeartifact.us-west-2.amaz
onaws.com/pypi/***/' does not have support for the following
features: --skip-existing and is an unsupported configuration

There is no workaround AFAICT as the twine version nor the --skip-existing flag is configurable.

Details on the twine change are here: https://twine.readthedocs.io/en/stable/changelog.html

Remove hacks that support --skip-existing for indexes other than PyPI and TestPyPI.

To date, these hacks continue to accrue and there have been numerous issues with them, not the least of which being that every time we update them, the paid index providers change things to break the compatibility we implement for them. Beyond that, these hacks do not work when text is internationalized in the response from the index provider.

[bhrutledge]

Is the fix to remove --skip-existing here?

https://github.com/cdklabs/publib/blob/aa230620b5037de7e9b125fd0cd4e34960854858/bin/publib-pypi#L39-L40

I tried to work around this by running pip install 'twine<6.2.0' before publib-pypi, but Twine was upgraded anyway because of the --upgrade option:

https://github.com/cdklabs/publib/blob/aa230620b5037de7e9b125fd0cd4e34960854858/bin/publib-pypi#L54

[mrgrain]

Happy to take a PR to make this option optional and/or an option to use a specific version of twine.

[mrgrain]

We can also try and rebuild the skip existing support in pulib. Maven is doing something similar.

[bhrutledge]

We can also try and rebuild the skip existing support in publib.

Something like this would be my vote. It seems like Twine and --skip-existing is an implementation detail that would be better to keep encapsulated (rather than exposing it as an option). This part of publib-ca feels related:

https://github.com/cdklabs/publib/blob/aa230620b5037de7e9b125fd0cd4e34960854858/src/codeartifact/staging/pip.ts#L28-L33

Aside: I used to be a maintainer of Twine, and I was the last person to touch some of the code that was removed.

[toxygene]

To elaborate, the Twine devs have stated:

Twine was and always will be PyPI first. https://github.com/pypa/twine/issues/1265#issuecomment-3258562261

AWS CodeArtifact implements only a subset of the PyPI repository API (https://docs.aws.amazon.com/codeartifact/latest/ug/python-compatibility.html), so Twine isn't guaranteed to work with it.

[mrgrain]

AWS CodeArtifact implements only a subset of the PyPI repository API (https://docs.aws.amazon.com/codeartifact/latest/ug/python-compatibility.html), so Twine isn't guaranteed to work with it.

Yeah. Coincidently AWS CodeArtifact is returning the correct HTTP status code (409) which Twine does check for. Except that the option was just deactivated for all non PyPI repos.

[antonu17]

We could also switch from twine to uv publish https://docs.astral.sh/uv/guides/package/#publishing-your-package

[gradybarrett]

We could also switch from twine to uv publish https://docs.astral.sh/uv/guides/package/#publishing-your-package

Seems like switching tools might be the right answer given twine's stance (pypi first). I respect it, but it means it's not the right tool for flexible publishing. Won't weigh in on the tool wars, other than to say that uv is compelling, even if it is VC-only backed atm.

[sigmavirus24]

https://github.com/pypa/twine/pull/1270 might also be of interest to folks here

[mrgrain]

Seems like switching tools might be the right answer given twine's stance (pypi first). I respect it, but it means it's not the right tool for flexible publishing. Won't weigh in on the tool wars, other than to say that uv is compelling, even if it is VC-only backed atm.

Not sure what alternatives we have. uv doesn't necessarily make it easier, e.g. it requires a config file for alternative repository urls. I also cannot tell if it supports attestations yet.

[gradybarrett]

Not sure what alternatives we have. uv doesn't necessarily make it easier, e.g. it requires a config file for alternative repository urls. I also cannot tell if it supports attestations yet.

Yeah, just sorta moves the problem around a bit. Poetry has similar config requirements. But the main thing is does it support diverse registries better than twine long-term and can the configs be reasonably abstracted.

I kinda think the tool shouldn't be exposed - like instead of twine env vars, there's env vars for like "PYTHON_REGISTRY_USERNAME", that can then be wrangled into whatever config or env var setup the underlying tool uses. Then you can swap tools at will. I'm less worried about npm, maven, and others. But the Python package tool ecosystem is just rough in my experience, so that isolation might be useful.

[tmokmss]

Hi I'm getting the below error on PyPi publish task because of the canges from https://github.com/cdklabs/publib/pull/1718. Workflow log

Run npx -p publib@latest publib-pypi
npm warn exec The following package was not found and will be installed: [email protected]
ERROR: Invalid requirement: "'twine<6.2.0'": Expected package name at the start of dependency specifier
    'twine<6.2.0'
    ^
Error: Process completed with exit code 1.

It seems the variable expansion is not working as expected.

$ packages="'twine<6.2.0'"
$ set -x
$ python3 -m pip install --user --upgrade $packages
+ python3 -m pip install --user --upgrade ''\''twine<6.2.0'\'''
ERROR: Invalid requirement: "'twine<6.2.0'"

https://github.com/cdklabs/publib/blob/a667541e075a88946c0c9c110064ed7acde3b1c7/bin/publib-pypi#L42-L54

[mrgrain]

@tmokmss Try again pls!

[tmokmss]

Confirmed it now successes (log). Thanks!

[woodruffw]

FWIW, you'll unfortunately run into similar issues with uv publish -- they made the (IMO correct) decision not to support --skip-existing from the get-go, because of exactly the kinds of problems that twine has had with third-party indices.

(uv publish's solution to all of this is to allow users to supply an index URL that can be checked for upload presence. However this isn't guaranteed to generalize super well either -- indices might not have publicly accessible index URLs corresponding to the upload URL, and they might only provide a small subset of the PEP 503 index API. They might also not show uploads immediately or until manually approved, meaning that existing distributions can't be checked on reasonable automated timescales. All in all, it's a pretty nasty problem that stems from a lack of a standardized upload API.)

[bhrutledge]

@mrgrain https://github.com/projen/projen/issues/4352#issuecomment-3401111273 says this has been resolved. Can you link to where/how that was done? Should this issue be closed?

[mrgrain]

We have pinned twine to a previous version. It's resolved from projen's perspective (it never really was a projen issue in the first place). There's still stuff to do in publib.