Add `Logging/Forwarding/Dropped` metric for log forwarding

[JcolemanNR]

Feature Description

There was one metric that was not implemented for the log4net forwarding EAP: Logging/Forwarding/Dropped

This was not implemented since a simple strategy could not be identified to track this metric. This metric should be re-evaluated before the GA release of log forwarding.

Describe Alternatives

The basic approach of taking a difference between the seen and sent metrics could result in incorrect calculations.

A more advanced, but accurate approach would involve modifying the PriorityQueue class used by the log forwarding feature to explicitly report how many items are pushed out of the priority queue.

Additional context

The feature definition from the internal engineering brief is somewhat ambiguous as well, and likely requires more discovery to properly implement this metric: UI wants to show clearly when we've excluded log lines to protect customer apps (and point to settings, other forwarding options, etc.)

For Maintainers Only or Hero Triaging this bug Suggested Priority (P1,P2,P3,P4,P5): P3 Suggested T-Shirt size (S, M, L, XL, Unknown): M

[angelatan2]

We may need to investigate the approach. The assumption of seen - sent = dropped. However, that may not be accurate for short period (when it's within the harvest cycle - 5 sec).

Potential solution - write the seen and sent before clearing out so it will be consistent over time. Review node agent implementation. It's ok if it's not accurate +/- 5 secs.

[workato-integration[bot]]

https://issues.newrelic.com/browse/NEWRELIC-3677

[workato-integration[bot]]

Jira CommentId: 170129 Commented by jcoleman:

[~cventura] pointed out that in addition to counting log messages that are discarded due to the size of the log event pool, we should also count the logs that are discarded because they don't contain a message or exception.

[workato-integration[bot]]

Work has been completed on this issue.