Feature flags on each alert message

[istreeter]

RDB loader sends alerts via a webhook to a listener, configured here. It is expected that every payload is an actionable alert, to be addressed by the pipeline operator.

There have been times when we've had alerts mis-firing, either due to bugs or due to known external problems. It would be helpful if every alert can be individually silenced, especially while this entire feature is still in an experimental phase.

Currently these are the places where the alerts are fired:

1. This info message for "Folder is already loaded"
2. This warning message if the HealthCheck feature cannot connect to the warehouse
3. These warning messages in the Folder Monitoring feature, if a folder is unloaded or corrupt
4. Another warning message in Folder Monitoring if there is any unexpected exception
5. A warning message if an exception is raised when loading a batch.
6. An error message and an exception is raised anywhere else, which causes the app to shut down.

Regarding points 5 and 6, there are several places where we explicitly raise an exception. Often we raise a LoaderError which is a sealed trait. E.g. here and here and here. But occasionally it is not an exception from the sealed trait:

• An IllegalStateException when something goes wrong initializing the manifest table
• A RuntimeException when we cannot read from S3.

I propose we first extend the LoaderError sealed trait to cover all possible explicit exceptions. (This includes removing LoaderError.RuntimeError, which is too generic). After that, we can have a config block which turns on/off the exception messages and warnings messages. Something like.....

"webhook": {
  "featureFlags": {
    "healthCheckWarning": true,
    "unloadedBatchWarning": true,
    "loadTimeoutError": true
    // etc
  }
}