Handle rate-limiting more gracefully

[PedroHLC]

> repoctl down firefox-nightly
Error: package "firefox-nightly" could not be found on AUR
> repoctl --debug down haxm-altea-git
Error: package "haxm-altea-git" could not be found on AUR
> ping aur.archlinux.org
PING aur.archlinux.org(luna.archlinux.org (2a01:4f8:160:3033::2)) 56 data bytes
64 bytes from luna.archlinux.org (2a01:4f8:160:3033::2): icmp_seq=1 ttl=47 time=231 ms
> ping -4 aur.archlinux.org
PING aur.archlinux.org (5.9.250.164) 56(84) bytes of data.
64 bytes from luna.archlinux.org (5.9.250.164): icmp_seq=1 ttl=47 time=230 ms

I'm pretty sure firefox-nightly will always be on AUR, but on random times, repoctl is not finding ANY package at all. I do have an hourly service downloading the same packages, but didn't find anything to understand/reproduce the error. This started +/- a week ago...

[cassava]

First of all, thanks for submitting an issue! Are you still having the problem? I wonder if the AUR query service was down at those times where it didn't work.

This file here contains the code for retrieving information from AUR: https://github.com/goulash/pacman/blob/master/aur/aur.go#L206

If the problem occurs, it would be very useful for you to post the content of:

curl 'https://aur.archlinux.org/rpc.php?v=5&type=multiinfo&arg[]=firefox-nightly'

This should be a JSON with the search results.

[PedroHLC]

Hey, seems like there's a rate limit in AUR

{"version":5,"type":"error","resultcount":0,"results":[],"error":"Rate limit reached"}

Learned the hardway 🤓

Feel free to close this issue...

[cassava]

@PedroHLC, do you regularly run into this problem? At minimum I want to dump an error when this happens, but while I'm at that I was wondering if I should try to optimize queries somehow.

[cassava]

(Whenever repoctl queries AUR, it does so in 200 package searches. So if you do a repoctl list -a and you have a local repo size <=200, then it sends a single request. If you have 660 packages, it will need 4 requests, etc.)

[PedroHLC]

Hi @cassava, the thing is that I have to download almost 300 VCS hourly, using repoctl down as there is no other way of checking their versions. But when it got larger then 200 pkgs, "repoctl down" was failing to run, so I did a for-loop to exec "repoctl down" for each package, which was the cause of this issue, currently I've replaced it with a "xargs -L 200" .

[cassava]

Hmm, repoctl down shouldn't fail to run with more than 200 pkgs... thanks for the info! I'll see if I can reproduce that.

[AladW]

You can use a POST request to submit up to 5000 packages in one request. But the following statement is unclear:

I have to download almost 300 VCS hourly, using repoctl down as there is no other way of checking their versions

Wouldn't repoctl down only download the packages with the versions advertised on AUR? If so, for VCS packages, those tend to be behind the actual upstream version (it's ill-advised to bump the AUR package for every commit).

You can just run makepkg -od in that case and compare the resulting version against your local repo.

[PedroHLC]

Hi @AladW

You can just run makepkg -od in that case and compare the resulting version against your local repo.

Yeah, in the end, we just let makepkg fail. But we keep downloading those from AUR hourly, as different builders build it with no cache, and we want the latest revision.

Wouldn't repoctl down only download the packages with the versions advertised on AUR? If so, for VCS packages, those tend to be behind the actual upstream version (it's ill-advised to bump the AUR package for every commit).

I meant here that the repoctl down -u wouldn't work on those guys because of the reason you stated.