PkgServer.jl icon indicating copy to clipboard operation
PkgServer.jl copied to clipboard

Add endpoints to obtain files from within registry tarballs

Open bauglir opened this issue 8 months ago • 4 comments

I have been working on adding package servers as a "datasource" to the dependency manager Renovate (renovatebot/renovate#29623). This seems like a nice approach as it ensures the dependency manager uses the same mechanism as (most) Julia clients and is therefore more likely to have accurate data (i.e. instead of directly interacting with the underlying Git repository).

One of the things I have run into is that, as far as I can tell, I can only obtain the full tarball of a registry (at a particular "state") and need to handle all the extraction and parsing locally. Given that these tarballs are relatively big and we often only need one or two files from them, it'd be nice if package server had endpoints available "indexing" into the tarballs and handling the necessary extraction and caching in a centralized location. I'm not sure how well this plays with the infrastructure and architecture, but having this information available through specific endpoints would make it a lot more straightforward to write tools like Renovate against package servers.

Some additional notes as to why I consider relying on package servers a nicer approach over directly interacting with the underlying Git repositories:

  • The Git repositories can be hosted on arbitrary hosting platforms, so support would have to be added for all those hosting platforms (i.e. interacting with their individual APIs) in Renovate.
  • Package servers can be used to provide access to "private" packages for which access to the source repositories needs to be restricted (including the registry's repository). It'd be nice if a dependency management system could support these.

bauglir avatar Jun 26 '24 21:06 bauglir