arrow icon indicating copy to clipboard operation
arrow copied to clipboard
verified publisher icon apache

[Python][Packaging] Stop uploading nightly wheels to gemfury

Open raulcd opened this issue 1 year ago • 7 comments

Describe the enhancement requested

Once we start uploading nightly wheels to the scientific python channel on anaconda:

  • https://github.com/apache/arrow/issues/40216

we can drop uploading wheels to gemfury. We should also update any documentation about nightly wheels if any.

Component(s)

Packaging, Python

raulcd avatar Sep 02 '24 08:09 raulcd

I think we should probably keep it for some time on both, and at least start with documenting the new index url as soon as possible.

(for example at least keep until 18.0 is released, when the stable docs will have the updated url)

jorisvandenbossche avatar Sep 03 '24 12:09 jorisvandenbossche

We don't seem to have updated the documentation on the new index URL, we should try update it for the next release.

raulcd avatar Jan 15 '25 09:01 raulcd

I filed an issue to document the new index URL, https://github.com/apache/arrow/issues/45288, and just put up a patch for it, https://github.com/apache/arrow/pull/45289. So the new text can land in Arrow 20.0.0.

Do we have any sense of whether Gemfury is being used at all? I'm not totally sure we'd have to wait until the release after 20.0.0 to remove Gemfury from CI considering how we present it to users. Either way, I'm happy to follow-up with a PR removing Gemfury after https://github.com/apache/arrow/issues/45288 is resolved.

amoeba avatar Jan 16 '25 23:01 amoeba

Do we have any sense of whether Gemfury is being used at all?

It is certainly being used by a bunch of projects to test the dev version of pyarrow. For example https://github.com/search?q=https%3A%2F%2Fpypi.fury.io%2Farrow-nightlies%2F&type=code

jorisvandenbossche avatar Jan 17 '25 21:01 jorisvandenbossche

Good call on doing that search. If we just stop uploading to gemfury and those projects don't migrate, I guess they'll think they're testing against a recent PyArrow nightly when they're actually just getting the last version we published to gemfury which could be very out of date. Maybe we can do the following:

  1. Stop uploading nightly wheels to gemfury
  2. Post to the dev list (using careful language to make it clear the nightlies aren't an official release, etc, etc)
  3. Filter through that GitHub search and identify repos to file issues against to notify of the change and give them a cutoff date
  4. Disable/delete or otherwise remove the arrow-nightlies index so installs against it fail. This would be done

Is that too much? If it sounds good, I can take on 1, 2, and 3 if someone with credentials can do 4 when the time comes.

amoeba avatar Jan 17 '25 22:01 amoeba

@amoeba should we follow this up? I've submitted a couple of PRs to the biggest repositories (dask and python-bigquery). I think we should get rid of gemfury wheels upload, possibly after 21.0.0? Reasons:

  • Nightly wheels are not official released artifacts and we always add the warning to the documentation.
  • We have been uploading them on the scientific nightlies channel since 4th September 2024
  • We updated the documentation officially for 20.0.0

raulcd avatar Jun 23 '25 07:06 raulcd

Hey @raulcd thanks for the reminder and thanks for doing those PRs. Moving ahead with this for the 22.0.0 release sounds good. I can start in on this once the 21.0.0 release is out.

amoeba avatar Jun 25 '25 03:06 amoeba

Issue resolved by pull request 47470 https://github.com/apache/arrow/pull/47470

raulcd avatar Sep 03 '25 07:09 raulcd