Problem with website: down time, not responding, 500, 404 - often after new release

[amis92]

Current status: Status monitor: https://stats.uptimerobot.com/QAEwmS6XLG

There is a recurring problem with the website being unresponsive, not loading repository list, loosing some repositories for some time, responding with 404/500 errors. There might be mutliple causes making it mutliple issues, but every one of those has been ignored, as the website returns to correct functioning after some time, or after manual reset.

There seems to be a connection with a refresh operation performed after there's a new release detected. No other correlations were made.

---

This issue is an umbrella for all these reports, and should be treated as a parent for all those marked as duplicate of it, and be resolved only after all issues are fixed - although they may be separately opened and closed.

Issues already reported:

• #56
• #86
• #90
• #138
• #158
• #165
• #173
• #190
• #195
• #196
• #198
• #204

[BitPsycho]

Vanguard Repo was gone ... then back and now gone again ... sigh ... This is causing alot of unnecessary complaints from users asking me where the repo is ... AND undermines the faith in BS ... This is clearly a BUG and bugs can be catched don't they ?

[amis92]

This website is essentially an abandonware at this point. If you know any web developer happy to contribute, please share.

I'm not in position to fix this, and none of the other people around here are, AFAIK.

[bguyan]

whats the problem? is there a way i can see the code for the site? ::EDIT::

I came here from a link and didnt see where i was... ignore the stupid question..

[amis92]

@bguyan my guess is that the webapp uses up the GitHub API request limit and some repos "fall-off" the site simply because during refresh cycle they were the last ones and got the Rate-Limiting error.

[bguyan]

@amis92 Yeah I cant do too much today but ill create a few calls to the API and see what i get if its a problem with numbers we can look at paginating or breaking up the calls.

[amis92]

I've investigated some optimizations in #112 and I think the best approach would be to use the webhooks. It'd probably drastically drop the number of requests made.

[NistrumCain]

OK. to solve this problem we need to make sure we override the API default with a per-page listing.. at the moment we are hitting some sort of default max when we query the page..

so if you make this call

https://api.github.com/users/BSData/repos

you only get to F

but if you make this https://api.github.com/users/BSData/repos?per_page=500 you get them all.. i just dont know where those calls are being made?

[amis92]

It's in https://github.com/BSData/bsdata/blob/master/src/main/java/org/battlescribedata/dao/ReleaseService.java#L41-L56 - it seems the call is already being paged.

[amis92]

Soo this is where the call is made. It seems to be paged, but I don't have the libraries it's using right now, so I don't know what do these different calls actually do.

https://github.com/BSData/bsdata/blob/41af6dbb415918dd9d1d074f23c36d3fc05919f4/src/main/java/org/battlescribedata/dao/ReleaseService.java#L41-L56

[NistrumCain]

so this java backend is the system that hands off to battlescribe? i could make the request in the frontend just though github but im guessing the java is the bit that talks to BS?

[amis92]

Yeah. Java backend is a kind of a REST web API, and BattleScribe talks to it, as well as the frontend, but the frontend isn't necessary for BattleScribe to do anything, it's just a separate client. BattleScribe needs the backend to serve up repo list, repo details, provide repo index and allow downloading of separate files.

[NistrumCain]

ok, well ill get this spun up locally and see if i can force it to make a call that collects all the repos and if i can ill create a pr if i can?

[BitPsycho]

Thanks guys for all the digging ...

Seems to me that the issue is simply the default page size being set in the standard GIT api code ?

public static final int PAGE_SIZE = 100;

Then if no parameters are passed along the call is executed with this size

/** * Create paged request with default size */ public PagedRequest() { this(PAGE_FIRST, PAGE_SIZE); }

It seems to me that the BS release code only performs 1 paged call ... one of size 100 ... and that's it ...

Are there more then 100 repo's now ?

[NistrumCain]

Tha5s what I was thinking.. i dont know the java bits well enough to be confident in my changes if i do make one.

[BitPsycho]

Rechecked the code again and it's not the paging size issue ... I jumped 2 soon to conclusion.

The code I mentionnes simply sets up a requester with page size 100.

The actual last line - getAll - creates a paged iterator - with this default size - which seems to query GitHub per page size until nothing is returned anymore ( all default code ) so I assume this is not the issue ...

It would call the GitHub API page by page ( 100 entries per page ) until none are found anymore .. and then return that collection ...

So unless there are HTTP errors while doing so I don't see an issue with this particular bit of code.

[NistrumCain]

Yeah... that's where I got confused.. I can make the request myself no problems I just dont get why it's getting problems.. I have no idea how we trial an error this

[migol]

Well, I'm quite upset about this issue TBH, so... do we have any logs from the server? I have quite a bit of Java experience and might be able to help out.

[amis92]

@migol data site should be up right now.

Logs are scarce and unfortunately don't exactly help because of the system design.

There are a few issues we're aware of, and all of them have been considered wontfix for various reasons:

• GitHub API rate limits
• AppSpot background thread exhaustion
• Simply bad design of data caching+refresh

We're working on a new, much better system, decentralized via GitHub repositories themselves. You can track progress here: https://github.com/orgs/BSData/projects/3

However, the app developer is a bit blocking us, because iOS still doesn't support our new design.

[Mayegelt]

Hi @amis92 you closed the issue I put and referenced it to here. Not entirely convinced it is the same issue. As normal this resolves after a while. But this has been a release that showed as being released on appspot, but no files within. Then tried again several days later and same result, despite other ones like the 40k repo having successful releases.

[ghost]

@amis92 is there something we can do to avoid triggering these build failures?

[amis92]

I've redirected here because deep down the issue is that we have a poorly written system we cannot fix.

For an alternative attempt, please see the BSData/gallery project.

[nstephenh]

Question: As of today, is the status monitor top row gallery or appspot?

[amis92]

@nstephenh gallery is up whenever github is up because it's not a separate app. So yeah, any uptime checks here are for AppSpot only. Within the monitor the first line is the general "/repos" endpoint which provides a list of available repos (which might be empty, but the app is running), and the second is specifically checking 40k availability on AppSpot, which will not be available if it's not indexed. So if the first is red, no repos can be updated, if the second is red, 40k cannot be updated but others might be available.

[amis92]

Closing as abandoned due to repo archival: #269