Integrate MonitorablesStore functionality inside MonitorsManager

[dillu24]

Technical Story

When you have a multiprocessing system you have to watch out how many processes you are going to spawn. There are two reasons why you need to do this:

• If the number of processes becomes a lot bigger than the number of CPUs then the Operating System will spend most of its time context switching between processes which is time consuming
• Processes take a lot of memory, therefore if you spawn a lot of processes you may eventually run out of memory

If we only focus on the Monitors, currently we are creating a manager process for every type of monitorable and a monitor process for every monitorable. For example, suppose that the user added 4 cosmos nodes and 1 DokerHub repo for monitoring. On startup PANIC is going to start a ContractsMonitorsManager, NetworkMonitorsManager, NodeMonitorsManager, SystemMonitorsManager, DockerHubMonitorsManager and a GitHubMonitorsManager all in a separate process. In addition to this, PANIC will start 4 CosmosNodeMonitors and 1 DockerHubMonitor in a separate process. As a result we are creating a lot of processes which will portentially increase as the node operator adds more monitorables. At a larger scale we might end up having a slow system and/or run out of memory.

To solve this it is being proposed that we start reducing the number of processes by using a combination of processes and threads. We can start by first focusing on the Monitors, benchmark the implementation and if there is benefit we would incorporate these changes to other components. The idea is to have a single MonitorsManager which spawns a thread for each monitorable. As per the resources below, threads are more memory efficient and lightweight to handle. When implementing the threaded monitor we have two options:

1. To implement a long-lived thread which connects to rabbit once and performs work every 10 seconds.
2. To implement a thread whose work is to connect to rabbit, do work, disconnects from rabbit and terminates.

It is suggested that we perform implementation 1 because according to the RabbitMQ docs the rabbit server works better with long-lived connections

For this huge task to be completed we need to tackle the following:

• Implement the Strategy Pattern for every type of monitor to have a code-base of higher quality
• Implement a single MonitorsManager that is able to receive configurations and use the appropriate strategy to start a monitor in a separate thread based on the routing key
• Add the monitorables store functionality that we already have in present managers.
• Add the heartbeats functionality that we already have in present managers.

Therefore to easily handle this large change we will break the task described above into granular tickets.

The aim of this ticket is to integrate the already present MonitorablesStore functionality inside the MonitorsManager.

Resources:

• https://www.indeed.com/career-advice/career-development/multithreading-vs-multiprocessing
• https://timber.io/blog/multiprocessing-vs-multithreading-in-python-what-you-need-to-know/
• https://towardsdatascience.com/multithreading-multiprocessing-python-180d0975ab29
• https://refactoring.guru/design-patterns/strategy/python/example
• https://auth0.com/blog/strategy-design-pattern-in-python/

Requirements

The aim of the MonitorablesStore mechanism is for PANIC to provide a list of monitorables to the API. LIke this the API could easily determine what is being monitored by PANIC.

To achieve this we must perform the following:

• [ ] Connect the MonitorsManager to the MONITORABLE_EXCHANGE so that it can send the list of monitorables
• [ ] Implement a function which retrieves the list of monitorables and sends it to the MONITORABLE_EXCHANGE

Blocked by

#245 - #253

Acceptance criteria

Given: We access the redis container Then: We can see the correct set of monitorables being sent by the MonitorsManager

[dillu24]

@simplyrider Also suggested another approach for implementing the Monitors architecture:

To make sure that the system never runs out of memory and CPU processing power is kept to a low we must make sure that as the user adds more monitorables there aren't a lot of threads/processes running at the same time. A good approach to manage this is to implement a queue which manages how many threads execute at the same time. Therefore we can have the following:

1. We would have 1 MonitorsManager that has 1 thread listening for all type of monitorable configs and the other thread executing a batch of tasks from a multiprocessing queue every X seconds (X should vary according to how many monitorables we have).
2. When a configuration is received, the MonitorManager adds a task on a multiprocessing queue for each configuration. In this task we must specify the monitor strategy to execute, and the corresponding configuration.
3. Once 5 seconds elapse for the task thread, the task thread grabs Y tasks from the queue(Y should very according to how many monitorables we have), checks that their configurations were not updated by the user and if not it starts a monitor thread for that configuration. Afterwards it puts the same task to the end of the queue for another monitoring round later on.
4. The monitor process needs to connect to rabbit with a separate connection, retrieve data, send it and disconnect from rabbit.

Some Notes:

• It is best to stress test multiple approaches to see which once performs best. We need to test the multiprocessing approach (The one we have currently), multithreading approach, and the queue approach.
• Another important thing we must need to think about is the heartbeat mechanism. The queuing approach may require us to re-design the heartbeat mechanism, modify it for the MonitorsManager or remove it altogether if deemed not useful.

Some resources:

• https://medium.com/omarelgabrys-blog/threads-vs-queues-a71e8dc30156
• https://testdriven.io/blog/developing-an-asynchronous-task-queue-in-python/
• http://pymotw.com/2/Queue/
• https://developer.apple.com/library/archive/documentation/General/Conceptual/ConcurrencyProgrammingGuide/OperationQueues/OperationQueues.html#//apple_ref/doc/uid/TP40008091-CH102-SW1