Agent Feature Request: Send asset errors to backend as sensu events

[jspaleta]

Expected Behavior

When Asset fails to:

• download -> seeing this with rate limiting from github
• checksum verify -> config human error and/or security concern
• filter match -> config human error

send error message as "asset-error" event to backend server

Current Behavior

errors are logged in agent log.

Possible Solution

send error message as "asset-error" event to backend server as well as log.

Context

Lots of little corners when setting up configs for assets, it would be easier to diagnose errors flowed through to sensu backend as events to surface the problem so the asset configs can get fixed.
"Sensu monitor thy self"

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[agoddard]

We've seen this a bunch, mostly around misconfigured assets - but it's also come up with asset rate limiting (or hitting an asset server without a rate limit and getting a 5** back from the overloaded server). Complicating the issue is that asset failures often generate events due to check execution w/out corresponding plugin availability, so this could potentially somehow have to combine with those events (possibly add some metadata to the event payload for the check in question rather than create a separate event) though doing so could complicate filtering and handling.

A related issue is that asset actions are both very verbose in the logs at debug level, but absent from the logs at log levels above debug.