natlas icon indicating copy to clipboard operation
natlas copied to clipboard

Create a DLQ for Server to collect failures

Open 0xdade opened this issue 4 years ago • 0 comments

Is your feature request related to a problem? Right now if an agent submits a scan that the server can't parse, the server just throws a 500 and doesn't make any attempt to save the troublesome data. Instead, the agent can optionally (and not default) save the failed scans. Unfortunately having the agent save these scans means that to troubleshoot, the server owner has to control the agent that experienced the failure.

Describe the feature you'd like The server should have a dead letter queue (DLQ) where scans that fail to parse for one reason or another get dropped for further debugging. Sentry.io captures some data around exceptions if the deployment is configured to use it, but will often trim out a bunch of the larger fields (and rightfully so). By having the server save these failures, the agents don't ever have a need to save them and their logic can be further simplified.

Have you considered alternative ways to get this feature I don't have specific details of how to implement this. Just that the server should do it instead of the agents independently being responsible for saving failures.

0xdade avatar Jun 26 '20 06:06 0xdade