Existence of .pypirc causes silent failure?

[lorencarvalho]

Howdy!

Just experienced a strange issue, I had a .pypirc in $HOME and attempted to twine upload dist/* a project of mine. The contents of dist/ is a tarball and a wheel. Twine's output indicated that it had uploaded the wheel (but oddly not the tarball) however on pypi.org there was no new file!

After removing the .pypirc file, the same command uploaded both artifacts and they appeared on pypi.org as expected! Very strange...

The contents of my pypirc were:

❱❱❱ cat ~/.pypirc
[distutils] # this tells distutils what package indexes you can push to
index-servers =
  pypi
  pypitest

[pypi]
username: comradeloren

Anyways, clearly I was able to move past the problem, but I figured I'd drop a note here in case anyone else experiences the same :)

Thanks for looking!

[sigmavirus24]

I suspect you may have not found a bug with Twine but instead encountered cached assets from the CDN that fronts pypi.org

[lorencarvalho]

If I bungle a release and I want to remove the published version, re-package and re-upload, what avenue do I have? Should I just wait for the cache TTL to expire?

I think the "bug" (or perhaps a better way to describe it would be "unexpected behavior") I encountered was that twine outputs a progress bar suggesting that it did re-upload my artifacts, when in reality it doesn't appear to have. I'd much prefer a warning or error or "nothing changed, not uploading" kind of message. What do you think? I imagine you don't want to open things up to where folks are purposely busting the CDN cache....

[sigmavirus24]

Twine's behaviour is entirely dependent upon PyPI's. If the file is actually already there, we tell you as much. The behaviour you're proposing is already implemented. That said, we do rely entirely on PyPI here. Based on your pypirc, I'm not sure there's anything better Twine could have done in this situation. It's also unclear what version of Twine you are/were using.

Frankly, I offered a suspicion, but that was wrong of me. I should have pressed for more information because there's not enough here for me to be legitimately helpful.

[lorencarvalho]

hi @sigmavirus24

I hope my first paragraph didn't come off abrasive!! I was being 100% earnest in my question (regarding my path forward).

I also failed to mention that I think you are right and that my pypirc was a red herring, was more likely the CDN issue you described :)

I am using the latest twine (1.11.0).

So I guess to restate the issue, what I'm finding is that if there is some sort of error, twine outputs a progress bar (as if it is uploading my artifact) but it's not actually uploading (I think?). I actually have some repro steps!

Attempt an upload of any package with bogus credentials exhibits the behavior:

❱❱❱ twine upload dist/*
Uploading distributions to https://upload.pypi.org/legacy/
Enter your username: not_real
Enter your password:
Uploading hiss_repl-3.0.2-py3-none-any.whl
100%|██████████████████████████████████████████████████████████████████████████| 10.1k/10.1k [00:02<00:00, 3.72kB/s]

[sigmavirus24]

Aha! I now think that I fully understand what you're talking about.

This is a nuance that unfortunately, Twine can't control right this very second, but it does in a way roll-up into PyPI. So the way that we work right now is as follows:

• We collect your credentials and send an HTTP request using eager Basic Authentication with those credentials. As part of this request we upload the full contents of the file.

• If your credentials aren't valid, then we error out and stop.

• If your upload seems to have been successful, then we continue with the rest of the files in dist/* (in this example).

The progress bar isn't a lie, we are tracking the progress of sending the data to PyPI. The problem here is that we have no way of testing the authentication credentials you've provided us before attempting to upload a file.

I know this isn't a satisfactory answer, but it's the best we have right now. I suspect @di might know the appropriate place to file the feedback for this in Warehouse. In other words, having a way to test credentials may be good for user-experience. I can also imagine that it might be a poor security practice though.

[lorencarvalho]

That makes perfect sense!! Thanks for the explanation 🙂

[di]

This has improved a bit since originally filed: Twine will still show the progress bar, but should show a 403 Client Error message if the credentials are invalid.

Since #475 is similar, I'll copy my message from that issue here about why this is unlikely to change further:

While I don't see a real security issue with having a credential-validating endpoint (as this is basically what the upload endpoint already is) it's probably worth mentioning that the upcoming introduction of API keys could potentially allow for finely-scoped upload credentials (e.g., for a specific project/release/distribution) which would require unique credentials to be included with each request. Meaning, a given set of credentials may only be valid when uploading a given file, so it makes sense to just include them with each upload request.