Draft Release support in PyPI

[alanbato]

This PR aims to implement the functionality required to support draft releases in PyPI.

Drafts releases are intended to enable package maintainers to better control their release processes and pipelines, and giving an extra step for testing things out before a new release is available to the public. This is separate from pre-releases in the sense that pre-releases are for users to use and test, whereas draft releases are invisible to them, and are only inteded to be used by maintainers.

Previous discussions as to why a feature like this could be useful, can be found in #726 and in this thread.

Implementation details and Rationale

This implementations key change is the addition of a published attribute in the Release model, all the new functionality operates on this value. The published timestamp is set when a release is published, and its null value signals that the release is created but not published, so therefore it's a draft release. In other words:

• Using the current workflow, a release is created with both created and published set to the same timestamp. Everythig is as usual.
• A release is created, but not published, when the build client's POST request that creates it has a Is-Draft header, and it is set to True.
• An existing draft release can then be published through the PyPI web UI, which simply sets published to the current timestamp.

Aditionally, a Release now has a draft_hash calculated value, which will be explained later.

The differentiation between draft releases and published releases is done in two separate places: pypi.org and the Simple Index.

Draft releases in PyPI.org

Regarding the PyPI website, the main focus was to 1) filter out draft releases from showing in a project's page for regular users and to 2) add additional UI elements for maintainers to manage draft releases.

Part 1) was done by modifying the traversal logic of a project's release when looking a specific version in the following way:

• If the release version in the path includes a draft hash, compare it against that release's draft hash. If they're the same, show the page for the draft version.
• If the release version in the path does not include a draft hash, only match against published versions.

For example, if my-package's version 0.23.0 is a draft,

• Accessing https://pypi.org/project/my-package/release/0.23.0 would not resolve (404 page)
• But https://pypi.org/project/my-package/release/0.23.0--draft--<some-hash>/ would talke you to that release's draft preview, given that some-hash is that release's computed draft_hash. If the draft_hash is incorrect, it would not resolve.

As for Part 2), please see the Visual Demo with the annotated screenshots.

Draft releases and the Simple Index

In order to support the upload, download, and installation of the draft versions of packages for the project's release workflows, while still making those drafts invisible to regular users, we would've needed to make significant changes to the Simple Index API. Since this was a non-goal for the project, an alternative solution was used, in the form of a newly created Draft Index.

The Draft Index implements the same API as the Simple Index, with the key difference that it's isolated for every single draft release. This means that when you create a draft release for a project, PyPI: creates a separate index listing only that project, with only that version available for it.

In order to be able to install the draft version of your package, you need to pass down this index URL as an extra index to pip, by means of the --extra-index-url flag. This allows pip to fetch the wheels & sdist for your package's draft version from this special index, while fetching all its dependencies leveraging the Simple Index and its current infrastructure.

Similar to what we did with PyPI, we use the draft_hash value of a release to construct its draft index URL, in the format of: https://pypi.org/simple/draft/<draft_hash>/ The full pip command using this URL is shown on the release's page on PyPI for easy access

In order to make this draft_hash deterministic for the maintainer's automation needs, the current implementation computes an md5 hash on the concatenation of the project's name and version.

On the other side of the equation, in order to create and upload a draft release, build clients need to pass down the Is-Draft special header, and could let the package maintainer do so by accepting a --draft flag when uploading such a release. I made an example implementation of how this could work with Twine, and you can try it out yourself.

Other Details

• When uploading files for a draft release, files with the same name get overwritten, and new files are added to the release. This makes it possible to make changes to the project without bumping the version.
• Along the same lines, everytime a draft release is updated, it's metadata is refreshed to reflect the new changes.
• Since you can now create a release without making it available to the users, you are now able to do things like upload platform-specific wheels and publish it once the release is ready for general use.
• There were minor refactors made to the code in order to reduce the overall complexity of some parts of the logic with the addition of draft-version handling.
• The database migration needed to add the columns also specify how to backfill the values in order to make the deployment of these changes easier.

Request for Feedback

All feedback is welcomed and appreciated, and trying out how everything works by pulling in these changes and spinning up a local server is encouraged to get a feel of how this all work. I'll try to answer as many things as I can.

Aditionally, I'm specially looking for feedback on these things:

• The draft hash value and how it's calculated. Some people think it shouldn't be guessable by regular users, while others want it to be determinable by maintainers in order to automate their workflows. How can we achieve both? Should it be calculated, or should it be stored?
• The draft index. There's a concern that having isolated draft indexes hosted on PyPI might lead to the creation of "private indexes", which we don't want. How can we make it so these indexes are temporal and ephemeral?
• How draft releases are created. We settled on specifying a special header so the upload clients can signal PyPI that they inted to create a draft release. I'd like to know if this is reasonable from the upload client's side.
• What changes are needed in order for this PR to be "mergeable", I know the code and database migrations need to be merged in a certain way/order in order to phase out the change, and could use some advise on how to do it.

Finally, here's the:

Visual Demo

This is a simple project I created to showcase how the new features and workflow look like in PyPI.

When looking at the project homepage, there's nothing different than what we have now.

The release history only shows one version.

If we go to the Simple Index for this project, we can see there is only a wheel and a sdist for that one version.

But if we are logged in as a project maintainer and go to the Manage Project section, we see there's a draft release listed! The release table also has a new column, that list the "published at" date alongside the "created at" date, which may now be different.

Draft releases have a new option in the Option menu, Publish.

If we choose the Manage option on this draft release, we'll see that there's a "Publish release" section. This has the same functionality as the option in the previous menu.

If you click any of these two options, a popup appears to confirm the action. And if you do...

The release is published! And will now appear in the release history, the project's homepage, and the simple index/api.

If we wanted to create a new draft release, we can do so by passing the --draft option to twine. ( implementation) twine upload -r localpypi --draft ../simple-proyecto/dist/proyecto-0.0.3*

Navigating back to the release detail page, we can click the "view release" link to get to the preview page.

Looking at the preview, we can see some notable differences. The first one being the special pip command that will intall this draft version of the package by leveraging --extra-index-url and a special hash that will prevent regular others from installing this version without access to this information. The second difference is the "draft" badge, which you can click to return to the project home page.

The index url that we pass to pip is an actual simple-index compatible page that list only this project, and only these draft versions.

And now we can install said draft release! (this is an old screenshot where the draft release is version 0.0.2)

That's all folks! :notes:

[alanbato]

Ah, the tests are failing because of the db migration, I'm not super familiar with alembic so I'd appreciate a pointer as to how to fix it :)

[ewjoachim]

Congratulations for a non-trivial challenge of a PR :)

The draft hash value and how it's calculated. Some people think it shouldn't be guessable by regular users, while others want it to be determinable by maintainers in order to automate their workflows. How can we achieve both? Should it be calculated, or should it be stored?

Can we send it at upload time ? You upload a draft release, receive the draft hash in some way, and then you can automate things based on that ?

[brainwane]

@alanbato if you rebase this so it doesn't conflict with main and then make a fresh request for feedback on discuss.python.org I bet you will get some!

[alanbato]

Will do!

[brainwane]

Alan, the only merge conflict right now is translations/messages - could you fix?

[brainwane]

I requested further feedback.

[alanbato]

Fixed the conflicts in translations (I think theres room for improvement in our process there)

Also took @ewjoachim's advice and provided a default value of NOW() for the published attribute. So in the stage where we have migrated the database but not the code yet, packages that are released during this period are not mistaken for unpublished packages after the new code is deployed.

Thanks @brainwane for requesting further feedback, I'd love for more people to take a look at this and see what else can be improved :)

[alanbato]

Also, I think tests are failing because of the migration history is a bit wrong. I would appreciate assistance on that 😅

[di]

Sat down with @alanbato at the PyCon sprints and thought through a plan for what we need to do to get this mergeable

• Programmatic way to turn a draft release into a published release: One thing this feature is missing right now is a way for a user to tell PyPI that a draft release should be published without needing to log into PyPI and do this via the UI itself. Users publishing multiple releases across multiple CI systems probably want some way to determine once all their draft releases have been made, and tell PyPI to publish. This doesn't need to be included in this PR but we should have a plan to work towards it before merging.
• Ensure that twine is able to reconstitute and surface the URL where the draft release will live on PyPI, so users can use this for testing before publishing

[di]

One other thing: currently the make_draft_hash function is not taking into account the project's normalized name, because Project.normalized_name is a column property and not a proper column. We should eventually move towards making this use the normalized name for consistency.

[dstufft]

There is now PEP 694: Upload 2.0 API for Python Package Repositories, which has discussions on discuss.python.org which is relevant to this issue.

[warsaw]

Over in #16277 I've resolved all the merge conflicts that have crept in as main evolved.

I have not attempted to resolve any comments on this original PR, nor have I even run the tests on this branch yet (let's see what the CI status is). No promises that I haven't botched the merge conflict resolutions either.

I'm planning on seeing if I can help move this initiative forward, so I do plan on doing those things. I also based my branch on this one, so @alanbato gets all due credit for the original work.

[alanbato]

Oh, it's been quite a while! A thousand thank yous @warsaw, as I'm sure it was quite a lot of conlicts that came with time that you went through.

This PR was abandoned by me due to what I felt was not reaching a concensus on some of the pending decisions to about how this funcionality was going to be exposed to users, pending work on documenting (and then proposing a better version) of the PyPI API, and the discussion thread for this PR eventually went quiet too...

My free-time is not as abundant as it was 4 years ago, but I'd love to collaborate on this if we still think the community will benefit from such a feature :)

Let me know how I can help!

[warsaw]

@alanbato I'm happy to help, and it's actually part of my job now to pitch in! I think now that we have PEP 694 to work from, maybe we can make good progress again. I do have quite a few questions about the current PEP draft, so it's probably worth resolving those in conjunction with working on a PR. I greatly appreciate the work you've done so far!