cargo icon indicating copy to clipboard operation
cargo copied to clipboard

registry-index: Add release `date` to Index Format

Open reneleonhardt opened this issue 8 months ago • 23 comments

Potential use cases:

  • #15973
  • #5221

Problem

The registry-index JSON Schema contains the version in the vers key, but not the release date. It would be very helpful not having to request the API only for getting this information bit 😄

Coincidentally that date is the default sorting key for the web UI (notice the Sort by drop down).

Example JSON Lines: https://index.crates.io/2/cc

Please note that yanked is included in every line, even if it is false (default is not visualized in the web UI) for most versions in contrast to the release date.

Proposed Solution

Can the schema be extended by the release date which is visualized in HTML https://crates.io/crates/cc/versions

For example version 1.2.21 has been released on May 2nd, 2025:

"date": "20250502",

Please note that the release date of the first version is always shown in the web UI:

100 of 168 cc versions since December 16th, 2014
Image

Notes

No response

reneleonhardt avatar May 05 '25 15:05 reneleonhardt

@reneleonhardt could you describe your use case for needing to know the date?

Please note that putting a field in the index comes with costs. It will be filled in for every version of every package (at least from the current time on, maybe backfilled). This increases the size of the Index files, making updating of the index take longer. For this reason, the primary purpose of the Index is not to store general crate metadata but to provide what is needed for dependency resolution without further API calls or downloading a .crate file. #5221 could provide a motivation for doing so for dependency resolution but we likely couldn't fully resolve that request because we don't track the full history of yanking and unyanking to know the exact state of the crate for the given target-time.

epage avatar May 05 '25 16:05 epage

I don't want to persuade you if you don't have the resources for a new attribute 😄 I just saw that 99% of all lines were filled with a default yanked: false but the unique date information is missing.

My use case would be to parse Cargo.lock files and combine it with all available releases to display possible updates.

cargo update --verbose is not very intuitive. The first call only updates Cargo.lock while displaying no available versions at all. The second call only shows available versions of the form

Unchanged core-foundation v0.9.4 (available: v0.10.0)

And cargo update doesn't allow bumping Cargo.toml, so most of the time developers don't even know when a new minor update of a dependency has been released.

My tool shows for a typical go.mod for example:

Module Required Update Old New Debt Type
github.com/go-openapi/jsonreference v0.20.2 v0.20.5 2023-01-14 2024-03-04 415 days patch
github.com/go-openapi/jsonreference v0.20.2 v0.21.0 2023-01-14 2024-03-09 420 days minor
... ... ... ... ... ... ...
Technical Debt (29 years) Patch 6245 days (17 years) Minor 5022 days (14 years) Total 10417 days (29 years)

reneleonhardt avatar May 05 '25 17:05 reneleonhardt

I just saw that 99% of all lines were filled with a default yanked: false but the unique date information is missing.

Yes, we require the populating of fields that were initially required so the Index can be read by old versions of Cargo. See #14506 for more details.

cargo update --verbose is not very intuitive.

We are open to discussing improvements. The question of what is "intuitive" is dependent on the end-user and their goals though.

The first call only updates Cargo.lock while displaying no available versions at all.

Not quite sure what you are getting at with this.

And cargo update doesn't allow bumping Cargo.toml, so most of the time developers don't even know when a new minor update of a dependency has been released.

Yes, this is being tracked in #12425.

epage avatar May 05 '25 19:05 epage

Just an example that the release date is available in static dumps: https://crates.io/data-access#rss-feeds https://static.crates.io/rss/crates/cc.xml

But only the last 10 releases are being published. And there is no stable feed, so for example you can see the latest 0.8.2 but not 0.8.1 because there were 17 alpha versions between them, but most developers don't rely on prerelease versions: https://crates.io/crates/schemars/versions

FYI, if yanked is considered to be important (not optional), it's missing in the rss schema (i.e. 0.14.4 has been yanked a year ago). https://static.crates.io/rss/crates/hashbrown.xml https://crates.io/crates/hashbrown/versions

reneleonhardt avatar May 07 '25 09:05 reneleonhardt

The release date would be useful for gradual rollouts or for delaying updates as a precaution against rapidly-spreading supply-chain attacks: https://github.com/rust-lang/cargo/issues/15973 https://github.com/rust-lang/cargo/issues/11167

OTOH, https://crates.io/api/v1/crates/cc already includes a list of versions with dates (and even a history of yanks), so the complete data can be gathered if necessary. It just wouldn't be as easily usable by cargo due to usage policy/rate limits.

kornelski avatar Sep 28 '25 13:09 kornelski

I've also been thinking about dates to find out which of my dependencies haven't been updated in a long time, as a proxy for "possibly unmaintained". I suppose the static RSS feed might be good enough for that though.

cuviper avatar Oct 08 '25 20:10 cuviper

@rust-lang/crates-io any input on making published dates available in the Index so cargo can

  • avoid resolving to "too new" dependencies (#15973)
  • resolver dependencies to a date (#5221)
    • this does have more questions about yanked packages

The main questions I would have for this are

  • what do we name the field
  • what format do we store the date in

epage avatar Nov 12 '25 17:11 epage

any input on making published dates available in the Index

I'm all for it :)

what do we name the field

we use created_at in the database, but I'm not a big fan of that naming. I would suggest using something like pubtime (abbreviation for publish time, since the index format seems to prefer short keys).

what format do we store the date in

ISO8601 with UTC timezone (e.g. 2025-11-12T19:30:12Z)

Turbo87 avatar Nov 12 '25 18:11 Turbo87

I'm good with all of that.

To move this forward, we'd need support in crates.io and cargo and an FCP for each team before any non-reserseable decisions (e.g. stabilizing it within cargo).

We'll need to make sure we have tests for this. We could likely extend cargo-test-support to allow setting pubtime on .crates.

epage avatar Nov 12 '25 18:11 epage

if it helps, we can implement it on the crates.io side with a feature flag that we only turn on for the staging servers. might help with testing older cargo versions.

Turbo87 avatar Nov 12 '25 19:11 Turbo87

https://github.com/rust-lang/crates.io/pull/12315 implements it on the crates.io side :)

Turbo87 avatar Nov 12 '25 19:11 Turbo87

Do we want any index backfill, or does it happen automatically?

weihanglo avatar Nov 12 '25 19:11 weihanglo

the backfill would be semi-automatic on a per-crate basis. in other words: when you publish a new version of a crate, all old versions will automatically be backfilled. but we can also trigger the backfill manually if that is preferred. it should probably be coupled with a git index squash though.

Turbo87 avatar Nov 12 '25 20:11 Turbo87

Would be good to do a backfill eventually. I would say at least by the time we are stabilizing a feature using this. We may want to do it when we have an implementation for them so we can more easily allow people to test them in prep for stabilizing.

epage avatar Nov 12 '25 20:11 epage

https://github.com/rust-lang/crates.io/pull/12315 has been approved now, so we can merge it whenever we decide that to move forward with it. let me know if we should try it out on staging.

Turbo87 avatar Nov 14 '25 14:11 Turbo87

Go ahead. I'll get a PR together for Cargo's side.

epage avatar Nov 14 '25 14:11 epage

Posted #16265

epage avatar Nov 14 '25 21:11 epage

Tracking issue will be #16270

epage avatar Nov 17 '25 20:11 epage

I've enabled the pubtime feature on staging now: https://github.com/rust-lang/staging.crates.io-index/blob/master/cr/at/crates-staging-test-tb.

The field currently has microsecond precision, which was unintended. I'll reduce it down to second precision in a follow-up PR.

Turbo87 avatar Nov 18 '25 08:11 Turbo87

@weihanglo @epage can you let me know once you've confirmed that the new fields does not break older cargo versions? I'm happy to enable this on production once we have that confirmation :)

Turbo87 avatar Nov 24 '25 09:11 Turbo87

I'm happy to enable this on production once we have that confirmation :)

Should we have an FCP for stabilizing this before enabling it on production?

epage avatar Nov 24 '25 15:11 epage

when you say FCP, are you talking about an FCP in the RFC context or a different kind of FCP?

in any case, I'm not against it if you think that it would be useful :)

Turbo87 avatar Nov 24 '25 17:11 Turbo87

For any one-way door, the Cargo team holds an FCP (they aren't just for RFCs).

epage avatar Nov 24 '25 17:11 epage