Proposal for a simple canary test

[FedericoCeratto]

Implement a simple end-to-end tester that runs continuously:

1. Start probe-cli against fastpath.ooni.nu/_secret_canary
2. Serve a lorem ipsum text on /_secret_canary. If the client ipaddress is the same as the probe AND the current hour is even, serve a 1-line text to simulate blocking.
3. Wait for the process probe-cli -> collectors -> fastpath -> event detector -> RSS/Explorer to finish
4. Scrape the result and check if the canary is flagged as blocked or unblocked
5. Report the processing time and correctness to Prometheus. Alert on failure.
6. Repeat every 10 mins.

This should be really simple to implement and yet give us early warnings in case of failures in the "critical path" of most of OONI.

[bassosimone]

Xref: https://github.com/ooni/probe-cli/issues/37

[FedericoCeratto]

Update: @bassosimone suggests toggling blocking during the first half of every hour. The tool should be therefore named Crazy Ivan.