analytics.usa.gov icon indicating copy to clipboard operation
analytics.usa.gov copied to clipboard

Published 20 hours ago • verified publisher icon 18F

External data staleness monitoring

Open konklone opened this issue 9 years ago • 7 comments

We had our first server bug that delayed data updates for a time. We should have something that watches the real-time data feeds and alerts us if it sees stale data.

I suggest performing a HEAD request to https://analytics.usa.gov/data/live/realtime.json and looking for whether its Last-Modified header has become more than 5 or 10 minutes old.

konklone avatar Apr 09 '15 14:04 konklone

Do you have a preference for what tool to use in order to notify folks if data ever is more than 5-10 mins old?

ArcTanSusan avatar Dec 08 '15 06:12 ArcTanSusan

We haven't dived into it too much. Got any ideas?

konklone avatar Dec 08 '15 19:12 konklone

Since the dashboard is entirely front-end w/o a backend, we can try a number of email-sending JS tools such as https://github.com/eleith/emailjs.

ArcTanSusan avatar Dec 08 '15 20:12 ArcTanSusan

That's only appropriate for back-end use, as if it's used in the front-end it would expose the authentication credentials for the SMTP server.

konklone avatar Dec 08 '15 22:12 konklone

Hmm, this seems to be an ops task. Maybe a cron job that calls a python script to make requests to that url and check for the timestamp headers?

ArcTanSusan avatar Dec 08 '15 23:12 ArcTanSusan

Yep, I think it could be relatively self-contained. Ideally, it would be a little general-purpose open source monitoring app that we can host on a Heroku-like system, and which can take enough parameters to do this without much/any custom coding. I'm not sure if something like that exists, but it would be the ideal.

konklone avatar Dec 08 '15 23:12 konklone

@konklone: Thoughts on the nodeJS project https://github.com/iloire/watchmen?

ArcTanSusan avatar Dec 11 '15 08:12 ArcTanSusan