[WIP] [Logging] Support debug level log verbosity

[bryanchriswhite]

Objective

Add a "verbosity" degree-of-freedom to debug level structured logging output. In pursuit of maintainability, this SHOULD generalize such that it can support all CLI binaries (e.g. node, client, etc.).

Conceptually, debug log verbosity has a consumer side and producer side:

Consumer

As an end-user, I only want to see log out put which is either actionable or necessary to support/enable continued operation so that I don't waste time considering output that does not require action. Typically this excludes the debug level altogether but in some circumstances, I need to see additional, debug level, logs (e.g. troubleshooting). In these scenarios, I prefer to see the "minimal useful" (i.e. lowest degree of verbosity) debug level output, by default.

As a contributor, while debugging, I want to be able to exclude excessively "noisy", debug-level, log output so that I can more easily locate more relevant debug log lines and ignore irrelevant ones. I would prefer the lowest degree of verbosity by default but sometimes, I need to be able to consider higher degrees (e.g. validating assumptions, debugging edge-cases). I would also find it quite convenient to be able to specify the verbosity of a particular module independently of others.

Examples (illustrative only):

p1 Query Height
#=> 100

p1 -v Query Height
#=> Low verbosity debug-level log...
#=> 100

p1 -vv Query Height
#=> Low verbosity debug-level log...
#=> Medium verbosity debug-level log...
#=> 100

p1 -vvv Query Height
#=> Low verbosity debug-level log...
#=> Medium verbosity debug-level log...
#=> High verbosity debug-level log...
#=> 100

Per-module verbosity:

# P2P module only
p1 -v p2p
p1 -vv p2p
...

# P2P and Consensus modules only
p1 -v p2p,consensus
p1 -vv p2p,consensus
...

Producer

As a contributor, while writing code, I want to be able to include debug-level log output around mission critical and/or show-stoppingly significant parts of the code (e.g. maxing out OS resource limits). However, I don't want to have to be too concerned about the "noisiness" of the log output such that it requires real consideration in the debug logging code (i.e. debug logging should not need to be that complex).

Example (illustrative only):

logger.Debug().Msg("Low verbosity debug-level log")

... ¯\_(ツ)_/¯ ...

Origin Document

Goals

Consumer

• MUST be able to specify some optional degree of verbosity.
• MUST be able to couple to optional CLI flags (e.g. -v, -vv, etc.) while maintaining separation of concerns between library and CLI binary code.
• SHOULD default to the lowest degree of verbosity

Producer

• MUST be able to specify some degree of verbosity.
• SHOULD default to the least verbose degree.
• MAY support overriding the default degree of verbosity.

Related Code

... app/pocket/main.go ... need to add cobra integration, be consistent with [good example ?]

... LoggerModule ...

Open Questions

... how to signal to localnet? ... makefile? :roll_eyes:

Deliverable

• ...

Non-goals / Non-deliverables

• ...

General issue deliverables

• [ ] Update the appropriate CHANGELOG(s)
• [ ] Update any relevant local/global README(s)
• [ ] Update relevant source code tree explanations
• [ ] Add or update any relevant or supporting mermaid diagrams

Testing Methodology

• [ ] Task specific tests or benchmarks: make ...
• [ ] New tests or benchmarks: make ...
• [ ] All tests: make test_all
• [ ] LocalNet: verify a LocalNet is still functioning correctly by following the instructions at docs/development/README.md
• [ ] k8s LocalNet: verify a k8s LocalNet is still functioning correctly by following the instructions here

---

Creator: @bryanchriswhite Co-creator(s): @h5law

[Olshansk]

Adding valuable context from @bryanchriswhite so its not lost.