rfcs icon indicating copy to clipboard operation
rfcs copied to clipboard

Add "crates.io: Crate Deletions" RFC

Open Turbo87 opened this issue 1 year ago • 4 comments

Rendered

Turbo87 avatar Jun 20 '24 10:06 Turbo87

Maybe require all crate versions to be yanked for at least $long_time.

kornelski avatar Jun 20 '24 16:06 kornelski

How does this help me rename my crate that has users?

it doesn't. this will only be helpful for renames very early in a project's lifespan. for example: you discovered after two days that you wanted to use hyphens instead of underscores or vice versa. this won't help e.g. tokio rename itself to kyoto

Maybe require all crate versions to be yanked for at least $long_time.

This would likely have consequences for the 72 hour rule. npm allows deletion without prior deprecation (which I treat roughly similar to our yanking). if you made a publishing mistake, such a rule would probably prevent people from being able to quickly fix their mistake.

Turbo87 avatar Jun 21 '24 08:06 Turbo87

I'm mostly in favour of this change, but with one caveat: I think that there should be an explicit mention that, while deleting a crate or version will immediately remove it from public downloads, crates.io should reserve the right to retain the deleted version for some amount of time (maybe a few days after deletion) just as a safeguard from abuse.

72 hours is a pretty long time, and just thinking of how this feature could be abused, you could upload malicious content for that interval, then "get rid of the evidence" pretty quickly. Effectively, I think that it should be possible for someone to notice that abuse occurred, report it, and then have someone review the deleted content before it's immediately sent to the shadow realm. Doesn't have to be long, but it feels like a safeguard that should exist if this feature is added, and it should be one that's clarified does exist.


Additionally, I think that the current requirements beyond the 72-hour window are a bit strict, and would exclude some crates that I personally think might be worth including under this policy. For example, take this incredibly old crate: https://crates.io/crates/bow

I am the owner of this crate, but it relied on the long-fixed incoherent_fundamental_impls soundness hole, and is now broken permanently on all versions of Rust since… a while. Six years ago, I published an explicitly empty version to the crate including a message about how it's broken, and back before I really understood everything, I didn't yank the old, broken versions. In fact, I should probably go and do that now.

However, through a series of also-broken dependencies by me that I also haven't checked in years, is part of the swc-hosts crate, which is downloaded a handful of times a day. In fact, it was downloaded 54 times on Jun 11, so, my guess is that someone has it running in their CI somewhere, and this itself is massively inflating the number of downloads.

Now, as I said, I'm going to probably take a bit within the next few days to go and make sure all of these crates have a blank latest version that does nothing, and their past versions are all yanked. But doing this now, after 6 years, means I'm basically never going to cover that 100-download-per-month requirement unless I have another several years where I have no downloads at all. (Which will happen, but, yeah.)

My proposal for this is to modify the rules slightly: as an alternative to the 100 downloads requirement, allow crates whose latest version satisfies the 100 downloads requirement but whose versions which do not satisfy that requirement are over a year old and have never been downloaded more than 100 times a day. This allows those pesky CI outliers to be weeded out while still effectively meaning that nobody uses the crate.

clarfonthey avatar Jun 23 '24 05:06 clarfonthey

100 downloads is too low. I'm an unknown crate author and I've released a few crates with no dependents and no advertising of the crates at all. They've all easily reached 100 downloads in the first one or two days. Bots keep downloading old versions of crates too, so each new version essentially acts as a multiplier for bot downloads. Also, releasing multiple new versions of a crate in quick succession (e.g. to fix semver breaks) seems to exponentially increase the number of bot downloads. One of my crates got a thousand bot downloads in a day and currently has 3700 downloads, more than twice the downloads of the crate it depends on. To my knowledge nobody uses it yet.

maia-s avatar Jun 29 '24 15:06 maia-s

@clarfonthey as far as I know (part of) our backend storage already keep deleted files around for purposes such as this. deletions already exist, but at the moment they are restricted to admins.

@maia-s thanks for the feedback. as the RFC text mentions, these thresholds are not set in stone. if we notice that the thresholds are not sufficient we can always tweak the numbers.

It looks like so far no major blockers to such a change have been identified. I appreciate the feedback so far and the crates.io team will take it into consideration when deciding on this RFC :)

Over the past days, our support email address has received a significant number of deletion requests again from users that no longer think their crates are useful and would like to release those names to the public again. This currently leads to quite a bit of manual work for us, even if it is just responding with a rejection message.

To move this RFC one step further forward I propose to move it into FCP status, which will still be gated on reviews and support from the rest of the crates.io team for now:

@rfcbot fcp merge

Turbo87 avatar Jul 17 '24 10:07 Turbo87

Team member @Turbo87 has proposed to merge this. The next step is review by the rest of the tagged team members:

  • [ ] @JohnTitor
  • [x] @LawnGnome
  • [ ] @Rustin170506
  • [x] @Turbo87
  • [x] @carols10cents
  • [x] @eth3lbert
  • [x] @jtgeibel
  • [x] @mdtro

No concerns currently listed.

Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

See this document for info about what commands tagged team members can give me.

rfcbot avatar Jul 17 '24 10:07 rfcbot

@kornelski

Maybe require all crate versions to be yanked for at least $long_time.

@Turbo87

This would likely have consequences for the 72 hour rule. npm allows deletion without prior deprecation (which I treat roughly similar to our yanking). if you made a publishing mistake, such a rule would probably prevent people from being able to quickly fix their mistake.

I was thinking more on my culture concern in https://github.com/rust-lang/rfcs/pull/3660#discussion_r1647411989. The friction (and hidden nature of) deletion has helped to encourage a culture of yanking, rather than deletion. At least for myself, I've never considered the option of reaching out to the crates.io team to ask for a deletion.

I worry that offering deletion front-and-center on crates.io, or especially on the cargo command line, could end up shifting that culture in a negative direction. Looking back to when I was a new or intermediate community member, i could have seen myself clicking a Delete button, seeing that it was available.

I think requiring yanking (which is much more dramatic than the term deprecation, I can't speak to effects within npm) on the backend (without an elapsed time) would help ensure people see yanking as the primary / default action and would add a little bit of extra friction to discourage over use of deletion.

(didn't put this in that comment thread as I think this requirement is independent from adding something to the requirements for deleting versions)

epage avatar Jul 18 '24 13:07 epage

@JohnTitor @LawnGnome @Rustin170506 @jtgeibel ICYMI https://github.com/rust-lang/rfcs/pull/3660#issuecomment-2232942504 😉

Turbo87 avatar Aug 16 '24 08:08 Turbo87

:bell: This is now entering its final comment period, as per the review above. :bell:

rfcbot avatar Aug 16 '24 14:08 rfcbot

The final comment period, with a disposition to merge, as per the review above, is now complete.

As the automated representative of the governance process, I would like to thank the author for their work and everyone else who contributed.

This will be merged soon.

rfcbot avatar Aug 26 '24 14:08 rfcbot

Linking to a planned policy change in a thread comment, so that it's easier to find: https://github.com/rust-lang/rfcs/pull/3660#discussion_r1854135667

joshtriplett avatar Nov 23 '24 18:11 joshtriplett

planned policy change

  • see https://github.com/rust-lang/rfcs/pull/3731

Turbo87 avatar Nov 26 '24 12:11 Turbo87

FYI the revised RFC is now fully implemented. see https://github.com/rust-lang/crates.io/issues/9352 for more details.

Turbo87 avatar Jan 15 '25 16:01 Turbo87

Salute to all the experts here! I'm a Rust newbie who has gone through great lengths to scale China's "Great Firewall" (GWF) to be here. After seeing this question, I couldn't help but chime in with some humble suggestions.

I've forked the RFC repository, but I'm feeling a bit nervous about whether submitting a PR directly would be too presumptuous, so I'm leaving a comment here first to share my organized thoughts. The original proposal was written in Chinese and then translated using deepseek, so I hope it's understandable.

If it's convenient, I'd appreciate any guidance from the experts on how a newbie like me can contribute to Rust. Thank you again! Below are my small suggestions:

New Proposal

Prerequisites

Record crate status in crates.io with the following states:

pub enum CrateStatus {    
    /// Clients do not update this version but allow developers to specify it manually.
    PreRelease(TimeRange),    
    Release, // default status.
    /// This version should be prominently marked on both the website and client.
    LimitedRelease(SemVerRange),
    GoverningRelease(TimeRange),
    /// Clients do not update this version but allow developers to specify it manually.
    PreDelete(TimeRange), 
}

These statuses will support the deletion mechanisms below.

Deletion Mechanisms

1. Author-Initiated Deletion

  • Crates with version 0.1.x published less than 8 hours ago.

2. Conditional Deletion

  • Long-unmaintained crates with version 0.1.x.

3. Manual Deletion

  • Crates with critical issues and no active maintainers.

Rationale

0.1.x and 8h

Crates with version 0.1.x on crates.io are typically experimental. Authors consider them unsuitable for production use, aligning with SemVer conventions. Most authors adhere to this practice.

If a crate was published for learning purposes, authors often wish to delete it after achieving their goals. However, current policies prevent such deletions. Thus, providing a time window (e.g., 8 hours or longer) before indexing these versions is essential.

Such crates should initially have the status PreRelease(0h..8h). During this window, authors may freely delete them. After expiration, the status automatically transitions to Release.

Automated Deletion

Due to historical policies, many 0.1.x crates on crates.io are unmaintained and occupy desirable names (e.g., reqwest vs. request). This creates confusion and technical debt. New developers should not need to navigate such legacy issues.

A scheduled task will check 0.1.x crates in Release status and set their status to GoverningRelease(0d..90d). If no maintenance occurs within 90 days, the status transitions to PreDelete(0d..3d), and authors are notified via email. If unresolved after 3 days, the crate name enters a pre-deletion state on crates.io, allowing others to claim it.

Once a new crate uses the name, the system permanently deletes the old crate.

Manual Deletion

For critical issues in active crates, authors may request deletion by setting the status to PreDelete(0d..3d). After submitting a fixed version, authors can delete the problematic version at an appropriate time.

Direct deletions may impact dependent projects. Client tools must warn users about deleted versions and automatically upgrade to fixed versions. For abandoned crates with critical issues, the crates.io team may manually set the status to PreDelete(0d..60d). If the original maintainer returns during this period and provides a fix, only the problematic version is deleted.

Caching Considerations

Testing crates have limited impact and require no special handling. For widely used crates, client tools must support version deprecation warnings and automatic upgrades to fixed versions.

Renaming Proposal

The LimitedRelease status can facilitate name swaps between crates. For example, renaming reqwest(0.11.0) to request(1.1.1) would follow this process:

  1. The request author transfers ownership to reqwest and adopts a new name (e.g., support_community_development).
  2. The reqwest author accepts the name request.
  3. The system sets reqwest to LimitedRelease(0..0.11.0).
  4. The system sets request to LimitedRelease(2.0.0..). Name changes are breaking changes and require a major version bump.
  5. The system sets request to LimitedRelease(0..1.1.1).

Clients resolving dependencies for request within 0..1.1.1 will use the original crate, while versions >=2.0.0 will resolve to the renamed reqwest crate. This preserves backward compatibility and resolves legacy naming conflicts.

Drawbacks

Significant coding effort is anticipated.

mihudale avatar Feb 20 '25 03:02 mihudale

I've forked the RFC repository, but I'm feeling a bit nervous about whether submitting a PR directly would be too presumptuous, so I'm leaving a comment here first to share my organized thoughts. The original proposal was written in Chinese and then translated using deepseek, so I hope it's understandable.

Yeah, commenting on a silghtly-related merged RFC is not the best place to get visibility and feedback on your new idea 😅 However, the Rust Internals forum is a great place for that -- are you able to access that?

carols10cents avatar Feb 20 '25 15:02 carols10cents

are you able to access that?

Yes, it's easier than visiting github.com. I'll go there and repost. Thanks a lot @carols10cents~

mihudale avatar Feb 20 '25 19:02 mihudale