Feature request: add an endpoint that returns perl tarball urls in plain text format

[skaji]

metacpan has an endpoint that returns perl tarball urls in json format: https://fastapi.metacpan.org/v1/release/versions/perl

It's very helpful, thanks.

On the other hand, the json format is not very easy-to-use by shell scripts. In fact, I'm thinking about creating a perl-installer in shell script.

So it's nice metacpan has an endpoint that returns perl tarball urls in plain text format.

My expected response is something like:

5.31.6     \t https://cpan.metacpan.org/authors/id/B/BI/BINGOS/perl-5.31.6.tar.gz \n
5.30.1     \t https://cpan.metacpan.org/authors/id/S/SH/SHAY/perl-5.30.1.tar.gz \n
5.30.1-RC1 \t https://cpan.metacpan.org/authors/id/S/SH/SHAY/perl-5.30.1-RC1.tar.gz \n
5.31.5     \t https://cpan.metacpan.org/authors/id/S/SH/SHAY/perl-5.31.5.tar.gz \n
5.31.4     \t https://cpan.metacpan.org/authors/id/C/CO/CORION/perl-5.31.4.tar.gz \n
5.31.3     \t https://cpan.metacpan.org/authors/id/T/TO/TOMHUKINS/perl-5.31.3.tar.gz \n
5.31.2     \t https://cpan.metacpan.org/authors/id/S/SH/SHAY/perl-5.31.2.tar.gz \n
5.31.1     \t https://cpan.metacpan.org/authors/id/E/ET/ETHER/perl-5.31.1.tar.gz \n
5.31.0     \t https://cpan.metacpan.org/authors/id/X/XS/XSAWYERX/perl-5.31.0.tar.gz \n
5.30.0     \t https://cpan.metacpan.org/authors/id/X/XS/XSAWYERX/perl-5.30.0.tar.gz \n
...

Additionally, it would be nice we can change the response with some query parameter:

• Returns stable perl tarball urls only (say, with query ?version=stable)

  5.30.1 \t https://cpan.metacpan.org/authors/id/S/SH/SHAY/perl-5.30.1.tar.gz \n
  5.30.0 \t https://cpan.metacpan.org/authors/id/X/XS/XSAWYERX/perl-5.30.0.tar.gz \n
  5.28.2 \t https://cpan.metacpan.org/authors/id/S/SH/SHAY/perl-5.28.2.tar.gz \n
  5.28.1 \t https://cpan.metacpan.org/authors/id/S/SH/SHAY/perl-5.28.1.tar.gz \n
  5.28.0 \t https://cpan.metacpan.org/authors/id/X/XS/XSAWYERX/perl-5.28.0.tar.gz \n
  5.26.3 \t https://cpan.metacpan.org/authors/id/S/SH/SHAY/perl-5.26.3.tar.bz2 \n
  5.26.2 \t https://cpan.metacpan.org/authors/id/S/SH/SHAY/perl-5.26.2.tar.gz \n
  5.26.1 \t https://cpan.metacpan.org/authors/id/S/SH/SHAY/perl-5.26.1.tar.bz2 \n
  5.26.0 \t https://cpan.metacpan.org/authors/id/X/XS/XSAWYERX/perl-5.26.0.tar.bz2 \n
  ...
  

• Returns a specific version's perl tarball url (say, with query ?version=5.26.0)

  5.26.0 \t https://cpan.metacpan.org/authors/id/X/XS/XSAWYERX/perl-5.26.0.tar.bz2 \n
  

• Returns the latest perl tarball url (say, with query ?version=latest)

  5.30.1 \t https://cpan.metacpan.org/authors/id/S/SH/SHAY/perl-5.30.1.tar.gz \n
  

[oalders]

@skaji are you asking for an entirely new endpoint or would it work if the current endpoint understood an Accept header withtext/plain?

[skaji]

Both are fine with me.

In case of not creating new endpoint, I think some query parameter (eg: format=tsv or format=txt) is better than Accept header.

[skaji]

Currently https://fastapi.metacpan.org/v1/release/versions/perl returns not only version and download_url, but also total, took, authorized, etc.

If we use the same url for text format responses, I think it should returns the same information as json format.

My request is to return versions and tarball urls only. So if you accept my request, a new endpoint is desirable, I think.

[oalders]

A new endpoint would be fine with me. Are you looking to create it or would you like one of us to set this up? We are meeting in a couple of weeks for meta::hack and this would be easy to do.

[skaji]

We are meeting in a couple of weeks for meta::hack

Sounds great!

Are you looking to create it or would you like one of us to set this up?

I'm not familiar with metacpan api's conventions, so I would be happy if metacpan team create it.

On the other hand, if you are willing to help me implement this endpoint, I will work on it and create a pull request! For now, I don't know where to write the dispatch logic (changing the responses according to query parameter stable/5.26.0/latest as mentioned in the first comment).

[oalders]

Ok, so I'm going to try to get it done on our end. If we don't get it started by the time meta::hack ends, I will circle back to you with some instructions on how to get started. :)

[oalders]

@PatrickCronin is interested in taking this one on.

[PatrickCronin]

I got my development environment set up and dove in. Soon after I was talking to @rafl who pointed out a couple key considerations:

1. It seems like the jq command line utility should be able to do this, and
2. Implementing a new endpoint means it would need to be maintained long term, so the benefits of the new endpoint would need to balanced against the cost of maintaining it. In particular, if jq can do this, then we don't have do.

So I stopped coding and set off to see if I could learn enough jq to produce the desired output, and I think it looks promising. If it's decided that there's actually more value in implementing the endpoint, then we can do it, but I first wanted to bring the simpler solution back for consideration. So without further ado:

jq is well-behaved

We'll use jq to produce your expected response to the unfiltered request:

# Use as part of a pipeline:
> curl -s https://fastapi.metacpan.org/v1/release/versions/perl | \
	jq -r '.releases[] | [ .version, .download_url ] | @tsv'

or

# Reduce API hits for testing:
> curl https://fastapi.metacpan.org/v1/release/versions/perl \
   -o releases.json
> jq -r '.releases[] | [ .version, .download_url ] | @tsv' \
   releases.json

Produces:

5.031006    https://cpan.metacpan.org/authors/id/B/BI/BINGOS/perl-5.31.6.tar.gz
5.030001    https://cpan.metacpan.org/authors/id/S/SH/SHAY/perl-5.30.1.tar.gz
5.030001    https://cpan.metacpan.org/authors/id/S/SH/SHAY/perl-5.30.1-RC1.tar.gz
5.031005    https://cpan.metacpan.org/authors/id/S/SH/SHAY/perl-5.31.5.tar.gz
5.031004    https://cpan.metacpan.org/authors/id/C/CO/CORION/perl-5.31.4.tar.gz
5.031003    https://cpan.metacpan.org/authors/id/T/TO/TOMHUKINS/perl-5.31.3.tar.gz
5.031002    https://cpan.metacpan.org/authors/id/S/SH/SHAY/perl-5.31.2.tar.gz
5.031001    https://cpan.metacpan.org/authors/id/E/ET/ETHER/perl-5.31.1.tar.gz
5.031000    https://cpan.metacpan.org/authors/id/X/XS/XSAWYERX/perl-5.31.0.tar.gz
5.030000    https://cpan.metacpan.org/authors/id/X/XS/XSAWYERX/perl-5.30.0.tar.gz
...

Filter for only stable maturity

> jq -r '.releases[] | .maturity="stable" | [ .version, .download_url ] | @tsv' \
   releases.json

5.031006    https://cpan.metacpan.org/authors/id/B/BI/BINGOS/perl-5.31.6.tar.gz
5.030001    https://cpan.metacpan.org/authors/id/S/SH/SHAY/perl-5.30.1.tar.gz
5.030001    https://cpan.metacpan.org/authors/id/S/SH/SHAY/perl-5.30.1-RC1.tar.gz
5.031005    https://cpan.metacpan.org/authors/id/S/SH/SHAY/perl-5.31.5.tar.gz
5.031004    https://cpan.metacpan.org/authors/id/C/CO/CORION/perl-5.31.4.tar.gz
5.031003    https://cpan.metacpan.org/authors/id/T/TO/TOMHUKINS/perl-5.31.3.tar.gz
5.031002    https://cpan.metacpan.org/authors/id/S/SH/SHAY/perl-5.31.2.tar.gz
5.031001    https://cpan.metacpan.org/authors/id/E/ET/ETHER/perl-5.31.1.tar.gz
5.031000    https://cpan.metacpan.org/authors/id/X/XS/XSAWYERX/perl-5.31.0.tar.gz
5.030000    https://cpan.metacpan.org/authors/id/X/XS/XSAWYERX/perl-5.30.0.tar.gz
...

Filter for a specific version

> jq -r '.releases[] | .version="5.26.0" | [ .version, .download_url ] | @tsv' \
   releases.json

5.26.0  https://cpan.metacpan.org/authors/id/B/BI/BINGOS/perl-5.31.6.tar.gz
5.26.0  https://cpan.metacpan.org/authors/id/S/SH/SHAY/perl-5.30.1.tar.gz
5.26.0  https://cpan.metacpan.org/authors/id/S/SH/SHAY/perl-5.30.1-RC1.tar.gz
5.26.0  https://cpan.metacpan.org/authors/id/S/SH/SHAY/perl-5.31.5.tar.gz
5.26.0  https://cpan.metacpan.org/authors/id/C/CO/CORION/perl-5.31.4.tar.gz
5.26.0  https://cpan.metacpan.org/authors/id/T/TO/TOMHUKINS/perl-5.31.3.tar.gz
5.26.0  https://cpan.metacpan.org/authors/id/S/SH/SHAY/perl-5.31.2.tar.gz
5.26.0  https://cpan.metacpan.org/authors/id/E/ET/ETHER/perl-5.31.1.tar.gz
5.26.0  https://cpan.metacpan.org/authors/id/X/XS/XSAWYERX/perl-5.31.0.tar.gz
5.26.0  https://cpan.metacpan.org/authors/id/X/XS/XSAWYERX/perl-5.30.0.tar.gz
...

Return the latest perl tarball

> jq -r '.releases | sort_by(.version) | .[-1] | [ .version, .download_url ] | @tsv' \
   releases.json

v5.8.1  https://cpan.metacpan.org/authors/id/J/JH/JHI/perl-5.8.1-RC1.tgz

I should note that this unexpected result is correct from an asciibetical perspective, which you could see by running:

> jq -r '.releases | sort_by(.version) | .[] | [ .version ] | @tsv' \
   releases.json | uniq

While it's not perfect for this particular query, a bit more massaging could do the trick.

[skaji]

@PatrickCronin Thank you for working on this feature request.

Implementing a new endpoint means it would need to be maintained long term, so the benefits of the new endpoint would need to balanced against the cost of maintaining it.

I quite agree.

Well, I should explain why I would like to have this endpoint in detail.

We have some tools that helps us install perls. Among them, perlbrew and perl-build (plenv) are famous, I think. They are written in perl. Unfortunately, these days, we cannot assume that there already exists system perl before installing perl. In fact, Apple announced that scripting language runtimes including perl wouldn't be available by default in future versions of macOS.

So I think it is useful if we have a perl installation tool written in shell script. And actually, I'm working on it.

There are some difficulties if I use the current endpoint https://fastapi.metacpan.org/v1/release/versions/perl from that perl installation tool:

• The current endpoint returns releases in json format, but json is not easy-to-use by shell scripts. Yes, if you had jq, then json would be your friend. But I want to minimize requirements for that perl installation tool, and I don't want jq to be a requirement.
• The current endpoint does not support "sort releases by versions". It's not easy to sort releases by version in shell script. Sorting releases requires some code like https://github.com/tokuhirom/Perl-Build/blob/master/lib/Perl/Build.pm#L30-L34

So I want metacpan to have an endpoint that returns perl tarball urls in plain text format.

[oalders]

I had a chat with @PatrickCronin and he will not have the bandwidth to complete this work in the near future. So I'm opening this up to anyone who would like to implement this. A big thank you to @PatrickCronin for the time he has already spent on this! ❤️

[mickeyn]

The main issue I see is with the querying method not returning a hash (because of the way the framework works).

Filtering by a CSV versions param to filter specific versions is an easy addition to that endpoint.

For the plain mode - I can suggest a quick & simple implementation of a solution in the form of { releases: "text" }:

>> curl 0:5000/release/versions/Plack-Middleware-Assets?plain=1
{
   "releases" : "0.0.2\thttps://cpan.metacpan.org/authors/id/P/PE/PERLER/Plack-Middleware-Assets-0.0.2.tar.gz\n0.0.1\thttps://cpan.metacpan.org/authors/id/P/PE/PERLER/Plack-Middleware-Assets-0.0.1.tar.gz"
}

>> curl -s 0:5000/release/versions/Plack-Middleware-Assets?plain=1 | jq -r '.releases'
0.0.2	https://cpan.metacpan.org/authors/id/P/PE/PERLER/Plack-Middleware-Assets-0.0.2.tar.gz
0.0.1	https://cpan.metacpan.org/authors/id/P/PE/PERLER/Plack-Middleware-Assets-0.0.1.tar.gz

>> curl -s '0:5000/release/versions/Plack-Middleware-Assets?plain=1&versions=0.0.1' | jq -r '.releases'
0.0.1	https://cpan.metacpan.org/authors/id/P/PE/PERLER/Plack-Middleware-Assets-0.0.1.tar.gz

Is that an acceptable solution?

[oalders]

This is an elegant solution, but I think the issue is going to be that @skaji specifically wants to do this without jq, because he doesn't want to make it a requirement for this tool.

[mickeyn]

jq was just for visualization, I'm open to suggestions on the format to make it machine readable if it's not (I just followed the requested \t & \n).

[mickeyn]

@oalders: I made it pure plain-text, please check #988

[mickeyn]

my merged solution still didn't address the request for 'latest' - I'll handle that separately.