poetry icon indicating copy to clipboard operation
poetry copied to clipboard

poetry publish uses pyproject.toml instead of built artifacts for metadata

Open kapilt opened this issue 2 years ago • 5 comments

while working on a poetry plugin, poetry-plugin-freeze that adds a wheel post processing command for freezing dependencies specifications in metadata (Ala Requires-Dist) out of the lock file, we noticed an anomaly on the resulting packages when published via poetry publish. namely pip and friends would install the frozen dependencies per what's in the wheel, but poetry would not. further examination showed that poetry is using the pypi json api to determine dependencies, and when using poetry publish it creates those via contents of pyproject.toml instead of using the content of the artifact to upload (ie the file is ignored except for checksum in this method https://github.com/python-poetry/poetry/blob/master/src/poetry/publishing/uploader.py#L129). this isn't the case with other upload tools, like say twine which derive their metadata from the contents of the file. this unfortunately creates an issue for any overrides of package file metadata when using poetry to publish. it also creates some inconsistencies in workflow, building a wheel, changing pyproject.toml (say adding a dependency), and then uploading creates a mismatch on the artifact's metadata contents and pypi's metadata for the project.

kapilt avatar Mar 27 '23 14:03 kapilt

poetry is using the pypi json api to determine dependencies, and when using poetry publish it creates those via contents of pyproject.toml instead of using the content of the artifact

I think you misunderstand something. The pypi json api returns metadata that is derived from the first wheel that is uploaded, poetry does not have the power to make it do anything else

Edit: I'm already losing confidence in that. It's not a great API that allows the metadata to be provided twice: both in the file and separately in the body of the POST, is it? And I'm not 100% sure how warehouse behaves when both are present.

If twine does what you want at upload, then i'd suggest just using twine. I'm not sure that poetry was ever meant to be a general purpose publisher of wheels not built by poetry.

dimbleby avatar Mar 27 '23 15:03 dimbleby

warehouse will use the explicit post metadata over the file metadata, which is what causes the divergence.

also to be clear poetry is creating the wheel, and a poetry plugin is modifying it, but because poetry doesn't use the wheel metadata when uploading versus using pyproject.toml, and thus creating a plugin for this purpose isn't useful without replacing the upload command as well. however even sans plugins, this publishing behavior can result in inconsistencies per example workflow (build wheel, edit pyproject.toml, publishing wheel gets inconsistent metadata of file vs pyproject.toml at time of publish). unfortunately different tools seem to be differential wrt to using json api endpoint metadata vs taking their metadata from the file, so both need to be in sync, ideally the built artifact/wheel would be a source of truth when publishing to ensure that.

kapilt avatar Mar 27 '23 19:03 kapilt

I understand the position, I just don't necessarily agree with it. Once something has modified a wheel built by poetry - including a poetry plugin - clearly we're in new territory so far as poetry publish goes.

I don't think it would be wild for maintainers to choose not to go there. twine is a fine tool.

Just one person's opinion, maybe someone else will think this is worth supporting and put together a merge request.

dimbleby avatar Mar 27 '23 20:03 dimbleby

fwiw, as a result of this issues the plugin currently requires twine to publish.

also this seems worth fixing simply as a race condition around building wheels with poetry and updating pyproject.toml before publishing, which results in wheel mismatch to the uploaded metadata.

kapilt avatar Aug 20 '23 22:08 kapilt

if you find this worth fixing - then submit a merge request!

we have five months of reasons to suspect that no-one else does find it worth their time: but what you propose sounds harmless enough that I'd think a merge request stands a decent chance of succeeding.

dimbleby avatar Aug 20 '23 23:08 dimbleby