Add observability tools to the node application

[iduartgomez]

In order to have better observability and enable performance analysis (specially in async context) swap "log" for "tracing (include a compatible sink for log dependencies to get debug output from the dependencies) and use the open telemetry standard so we can plugin data to external systems (like prometheus or jaeger):

• https://crates.io/crates/tracing
• https://crates.io/crates/tracing-opentelemetry
• https://docs.rs/opentelemetry/0.17.0/opentelemetry/

[netsirius]

I have been researching and testing with the three libraries you mention.

Using all three together with opentelemetry-jaeger it is possible to achieve network traceability, both for a single node and between nodes.

In addition, the library mentioned allows exporting the traces to an external trace collector such as Jeager.

The configuration of the traces and the exporter can be configured in the application or by with environment variables.

I am going to prepare a diagram of the tractability of the application and do some tests on how to apply it throughout the network.

[netsirius]

I'm currently developing the solution, but I have created this draft https://github.com/freenet/locutus/pull/89

[iduartgomez]

We still need to polish this a little bit, so I am not closing the issue yet, but some good progress was done in #89

Things that I would like to do before closing this:

• figure out the remaining issues with not getting all the instrumentation working and reconstructing all the logs from different nodes (would be nice to have the latest)
• automatize the deployment of whatever we use within the repository (add some container definitions and deployment scripts)

(FYI @netsirius )

[github-actions[bot]]

Pivotal Tracker story: https://www.pivotaltracker.com/story/show/184207213

[iduartgomez]

Tracked in pivotal tracker