ecosystem-dashboard icon indicating copy to clipboard operation
ecosystem-dashboard copied to clipboard

Mechanism for getting "top dependents" of an npm package like js-libp2p

Open BigLep opened this issue 3 years ago • 9 comments

This isn't an issue with the Ecosystem Dashboard directly, but it's information I'd ideally hope to get out of the Ecosystem Dashboard if there isn't an alternative. I'm viewing this issue as an exploration which may lead into a feature request if there isn't another way to handle it.

Problem I want to solve: find the "top dependents" of an npm package like js-libp2p. I want to know this information to figure out consumers/dependents I should reach out to (e.g., for security disclosure, user feedback).

For example, I don't see a way to sort dependents by weekly downloads in the npm UI. Should I maybe use tools like https://www.npmjs.com/package/top-dependents or https://www.npmjs.com/package/npm-get-top-dependents can help?

Any expertise or suggestions @andrew are welcome.

BigLep avatar Sep 15 '22 20:09 BigLep

Although we don't currently track download numbers, we do have dependents of packages, for libp2p on npm, for example:

I've made some tweaks to those pages today to sort by our community score, shown on the right of the lists (a combination of stars, usage, activity levels etc) which will show the biggest users of a given package, including which version(s) of the package they currently depend upon.

I'm also currently working on highlighting dependent packages in the other projects I'm working on: https://packages.ecosyste.ms and https://repos.ecosyste.ms which should give similar, if less "internal vs external" data at a bigger scale (5 million packages and 32 million repos indexed)

andrew avatar Sep 19 '22 15:09 andrew

Thanks @andrew - this is great.

I think a key element here is how much we trust "community score". I guess the action here is for me to manually find some same dependents of libp2p in npm and spot some of the packages with higher downloads and check to see if they also have a high "community score".

The thing I want to avoid for example is disclosing to top "community score" dependents only to find out later that I missed some of the larger dependents (based on downloads) as that wouldn't be a good look for the project.

Thanks.

BigLep avatar Sep 19 '22 16:09 BigLep

New endpoints added to ecosyste.ms today:

  • https://packages.ecosyste.ms/registries/npmjs.org/packages/libp2p/dependent_packages
  • https://packages.ecosyste.ms/api/v1/registries/npmjs.org/packages/libp2p/dependent_packages

Currently no scoring or download tracking in ecosyste.ms yet but I can prioritze that, and have the ecosystem dashboard sync the relevant data from the packages api.

andrew avatar Sep 19 '22 17:09 andrew

Still working on download counts but you can now sort by most dependend upon packages which highly correlates with download counts: https://packages.ecosyste.ms/registries/npmjs.org/packages/libp2p/dependent_packages?order=desc&sort=dependent_packages_count

andrew avatar Sep 20 '22 15:09 andrew

Download stats have been implemented, I've synced the the dependencies of libp2p as an example, other packages will slowly update as the download data is synced over the next few days.:

  • https://packages.ecosyste.ms/registries/npmjs.org/packages/libp2p/dependent_packages?order=desc&sort=downloads
  • https://packages.ecosyste.ms/api/v1/registries/npmjs.org/packages/libp2p/dependent_packages?order=desc&sort=downloads

andrew avatar Sep 21 '22 15:09 andrew

Awesome @andrew - thanks a lot! This is very helpful.

Out of curiosity, how did you implement this (or I guess I can figure this out by looking at https://github.com/ecosyste-ms/packages).

Thanks again - feel free to close.

Maybe it's a separate issue to do linking out from the ecosystem dashboard to packages.eocystem.ms where someone can get more data?

BigLep avatar Sep 21 '22 16:09 BigLep

@andrew : one other thing:

npm says 107 dependents for libp2p: https://www.npmjs.com/package/libp2p?activeTab=dependents ecosyste.ms says 41: https://packages.ecosyste.ms/registries/npmjs.org/packages/libp2p/dependent_packages?order=desc&sort=downloads

Is that difference expected?

BigLep avatar Sep 23 '22 18:09 BigLep

@BigLep thanks for reporting that, there was a bug in my sql query, updated now and it actually finds more dependents (169), which I'm investigating as well.

andrew avatar Sep 26 '22 14:09 andrew

The increase in dependents in ecosystems is because npm only counts packages as dependent if the latest version is dependent, for example, the package @achingbrain/libp2p-gossipsub:

https://www.npmjs.com/package/@achingbrain/libp2p-gossipsub/v/0.13.9 is the latest and doesn't depend upon libp2p but the previous version 0.13.8 has a dev dependency on libp2p.

andrew avatar Sep 26 '22 14:09 andrew