GTFS-rt feed should meet a minimum HTTP request success rate

[barbeau]

Issue by barbeau Friday Apr 07, 2017 at 20:59 GMT Originally opened as https://github.com/CUTR-at-USF/gtfs-realtime-validator/issues/113

---

Summary:

For a few GTFS-rt feeds I've seen (MBTA being one), they will occasionally fail to return a result to the consumer when an HTTP request is sent to the GTFS-rt feed endpoint. In this tool, there should be a warning generated on each failure, and an error generated when the success rate falls below a certain threshold. For the summary section of each feed in the UI, and the total summary, we should also include a count of failures, and the current success rate.

For "Overview", UI should look like:

• Total HTTP requests: 50
• Total unique responses: 25
• Total request failures: 5
• Total request success rate: 90%

For "Feed - XYZ", the UI should look like:

• HTTP requests: 50
• Unique responses: 25
• Request failures: 5
• Request success rate: 90%

One open question for the GTFS-rt community is "what should this threshold be"?

For this tool, let's start with an error threshold of 5% - so if more than 5% of requests are failures, the tool will generate an error in the log, and will turn the text for "HTTP Request success rate:" red.

Steps to reproduce:

Start the tool

Expected behavior:

Show me a summary of HTTP errors, and log warnings/errors as appropriate for HTTP failures

Observed behavior:

HTTP failures are output to console, but not counted or shown in UI, and they do not generate warnings or errors.

[e-lo]

Relevant Best Practice:

The server hosting GTFS Realtime data should be reliable and consistently return validly-formatted protobuf-encoded responses. Fewer than 1% of responses should be invalid (protobuf errors or fetching errors).

[e-lo]

For this (and a few other rules, e.g. the ones related to id stability) I'm wondering if a monitoring-type functionality needs to be implemented (as opposed to a run-and-done)? This type of rule doesn't really provide insight unless the URL is pinged a bunch.

[barbeau]

Yes, the original idea for this issue was to have this feature as part of the webapp, so the batch validator wasn't considered. But it's possible that for some issues like this, and what I mentioned in https://github.com/MobilityData/gtfs-realtime-validator/issues/117#issuecomment-1072863509, that you'd want a monitoring service built into a command-line tool - something like a hybrid between the webapp and the current batch validator.