System monitor

[orbitalturtle]

This PR puts together a first draft of a system monitor for a Teos watchtower. It can be run once Elasticsearch and Kibana are running to produce a Kibana dashboard with several graphs displaying Teos data.

Would love some feedback if anyone has time!

I also have my eyes on the following improvements:

• Right now each time the program boots up, in order to gather the most recent log data, it wipes out the data that was entered into Elasticsearch entirely and refills it with the data from the log file. Would probably be better to track how many logs have been entered so far and when the program starts up, start where it left off.
• The data could be updated in real-time (maybe Logstash can help here? I'm not super familiar)
• I might be missing something, but right now it seems the regtest Teos log data and mainnet log data are mixed in to the same log file, which would muddle the data in the dashboard. So perhaps would be good to make separate files for each.
• Testing and exception handling are very minimal so far
• There's probably much more that can be graphed :-) esp. time series data

[sr-gi]

Supper excited about this 🚀 I'll give it a look asap.

[yahgwai]

Looks cool, always nice to see some dashboards :)

I think you might be able to do without your own custom loading code though. You can take two basic approaches to loading logs: handler or agent.

The handler approach is to add an additional logging stream. You already have a console and file stream:

self.f_logger.info(self._create_file_message(msg, **kwargs))
self.c_logger.info(self._create_console_message(msg, **kwargs))

so you would just add a third stream for elasticsearch. To do this you'd want to use a 3rd party lib as it would do a better job of keeping track of location (in case of crash), and avoiding blocking io. I dont work with python so I dont know how to assess the quality of the lib, but this seems ok: https://github.com/cmanaha/python-elasticsearch-logger

The agent approach is the one you've taken already, but you can use an existing agent instead of writing your own. Filebeat (https://www.elastic.co/beats/filebeat) will keep track of files and locations, and you can use logstash if you need transforms (https://www.elastic.co/logstash).

There's good discussion comparing the two approaches here (https://github.com/cmanaha/python-elasticsearch-logger/issues/44/) with the author python-elasticsearch-logger

[yahgwai]

You have potential problem with overlapping fields of different types. Elasticsearch wont accept the same field with a type different to it's mapping. So if you have a two logs like:

logger.info(message, testy="testy")
logger.info(message, testy=10)

the latter would be rejected.

I havent checked to see if there are any instances of this issue, just flagging it as potential problem.

[sr-gi]

Starting to check this finally, sorry for taking so long :(

[orbitalturtle]

Awesome thanks for the feedback @yahgwai and @sr-gi, I'll look into this over the weekend