phive is only scanning first page of GitHub release api

[lippok]

I want to install the old version 0.11.16 of phpstan, but the installation fails when the release is not in the local cache.

phive.xml:

<?xml version="1.0" encoding="UTF-8"?>
<phive xmlns="https://phar.io/phive">
  <phar name="phpstan" version="^0.11.16" installed="0.11.16" location="./tools/phpstan" copy="true"/>
</phive>

Output:

Phive 0.14.4 - Copyright (C) 2015-2020 by Arne Blankerts, Sebastian Heuer and Contributors Downloading https://api.github.com/repos/phpstan/phpstan/releases [ERROR] No matching release found!

The release is not listed at https://api.github.com/repos/phpstan/phpstan/releases but on https://api.github.com/repos/phpstan/phpstan/releases?page=2 but only the first page seems to be read.

[theseer]

That is apparently correct but not a trivial fix.

As github doesn't seem to provide an explicit API to find out how many releases actually exist, this information has to be deducted from the link header's last attribute.

Given the most common use case is likely to request the installation of a somewhat recent version, we probably do not want to always get the full list of all releases from all pages and with that have pointless requests hitting against the github api rate limit.

We thus need to implement a sort of streaming, lazy-loading kind of lookup that fetches additional release infos when no match is found and not all releases have been read yet.

That's completely different from the current implementation which consumes the downloaded json. Also, the downloader code doesn't yet support looking at the link header.

[theseer]

Starting phive 0.14.5 we're requesting 100 releases per "page". While that of course doesn't fix the actual issue, it at least mitigates the problem a little bit on the cost of some additional traffic.

I'm still unhappy with the github api ;)

[Xethron]

Hello @theseer, it seems like this problem is back.

I've been having problems downloading phpstan 0.12.99, which looks like it just dropped out of the last 30. Looking through the code, I see the ?per_page=100 is still in the GithubAliasResolver, however, it appears that this is not being used for phpstan, and instead, the URL in https://phar.io/data/repositories.xml is being used.

[theseer]

@Xethron Which version if phive are you using?

[Xethron]

Phive 0.15.0. I just managed to get it to work by changing

  <phar name="phpstan" version="^0.12.92" installed="0.12.99" location="./tools/phpstan" copy="false"/>

to

  <phar name="phpstan/phpstan" version="^0.12.92" installed="0.12.99" location="./tools/phpstan" copy="false"/>

I assume this bypasses the "alias" lookup, and then uses the GithubAliasResolver instead of the url in repositories.xml

[theseer]

Can you try using phive 0.15.1 and if the problem persists with it?

[theseer]

  <phar name="phpstan/phpstan" version="^0.12.92" installed="0.12.99" location="./tools/phpstan" copy="false"/>

I assume this bypasses the "alias" lookup, and then uses the GithubAliasResolver instead of the url in repositories.xml

Yes, that's a different resolver and I somehow managed to miss the other resolver when adding the per_page option to the URL.

[Xethron]

Just tested it with 0.15.1 and it works!

Thank you for your help, and thank you for the extremely fast replies :)

PS: Installing phive with curl -LsS -o phive.phar https://phar.io/releases/phive.phar still installs 0.15.0

[theseer]

Fixed that. Thanks.