github download badges invalid

[smallprogram]

Are you experiencing an issue with...

shields.io

🐞 Description

https://img.shields.io/github/downloads/smallprogram/OpenWrtAction/total

The Shields.io GitHub application has been authorized. However, the download badge shows invalid, and it only works for displaying the download count for a single release.

🔗 Link to the badge

💡 Possible Solution

No response

[jNullj]

We have a limit on API response, for example, at the default config the limit is 10MB. I confirmed that that response is larger then 10MB and that's what causes the invalid response as we cancel the request to Github's API once we reach that 10MB limit.

I will try to look into that later today.

[chris48s]

GitHub only expose download count at release asset level. They don't directly expose an aggregate count. The rest API doesn't provide a way to only request certain fields. We have to accept the full release object to get that one field we are interested in. In the case of this particular project, a single release object is very large because every release has hundreds of related assets.

One thing we could possibly look at doing is switching to the v4/GraphQL API (with GithubAuthV4Service). Using that, we might be able to write a query that only requests the ReleaseAsset.downloadCount field. That would return a much smaller response. I haven't tested this, but I think now that GitHub have dropped the max number of objects per page to 100 on the Rest API (even though the code still says per_page=500 GitHub now caps it at 100 objects - see https://docs.github.com/en/rest/releases/releases?apiVersion=2022-11-28#list-releases ) we could request the same number of releases in one query over the GraphQL API as we get back from the rest API. As I say, I have not checked it but worth a look.

In general, the 10Mb limit is there for a reason. It exists to keep performance and memory usage under control. If we have to download and parse >10Mb of data to serve 1 number, that is not a scalable badge. Even if we increased the limit, we probably couldn't serve this badge within camo's 4 second timeout. I am finding that just downloading https://api.github.com/repos/smallprogram/OpenWrtAction/releases?per_page=100 sometimes takes 5 or 6 seconds.

[smallprogram]

We have a limit on API response, for example, at the default config the limit is 10MB. I confirmed that that response is larger then 10MB and that's what causes the invalid response as we cancel the request to Github's API once we reach that 10MB limit.

I will try to look into that later today.

I think besides the 10MB limit, the problem is here

https://github.com/badges/shields/blob/28f075f9b21dc5801342083c44945b668cc6d269/services/github/github-downloads.service.js#L118-L135

You can change the logic to paginate requests, 100 per page, until there are no more, and then continue your reduce() operation. of course this is just my idea

[smallprogram]

GitHub only expose download count at release asset level. They don't directly expose an aggregate count. The rest API doesn't provide a way to only request certain fields. We have to accept the full release object to get that one field we are interested in. In the case of this particular project, a single release object is very large because every release has hundreds of related assets.

One thing we could possibly look at doing is switching to the v4/GraphQL API (with GithubAuthV4Service). Using that, we might be able to write a query that only requests the ReleaseAsset.downloadCount field. That would return a much smaller response. I haven't tested this, but I think now that GitHub have dropped the max number of objects per page to 100 on the Rest API (even though the code still says per_page=500 GitHub now caps it at 100 objects - see https://docs.github.com/en/rest/releases/releases?apiVersion=2022-11-28#list-releases ) we could request the same number of releases in one query over the GraphQL API as we get back from the rest API. As I say, I have not checked it but worth a look.

In general, the 10Mb limit is there for a reason. It exists to keep performance and memory usage under control. If we have to download and parse >10Mb of data to serve 1 number, that is not a scalable badge. Even if we increased the limit, we probably couldn't serve this badge within camo's 4 second timeout. I am finding that just downloading https://api.github.com/repos/smallprogram/OpenWrtAction/releases?per_page=100 sometimes takes 5 or 6 seconds.

In fact, the slow display speed is not the primary problem, but the error in counting the number of downloads is what needs to be solved urgently.

[chris48s]

For performance reasons, we don't write badges that paginate over an unbounded number of responses. We have to be able to return a response quickly. Ideally, we don't write badges that make more than one upstream request to serve a response. There are a few cases where we make more than one, but it is always a finite number.

That's why we only take into account the most recent N releases for "total" downloads.

[smallprogram]

I hope you geniuses can find a better way to solve this problem.