components-contrib
components-contrib copied to clipboard
dapr
Standardize behavior on component initialization failure
Currently, when components fail to initialize (for example because of a misconfiguration or connection error), the behavior is undefined and depends on the component.
Some components cause the sidecar to crash; others print a warning or error message; others do nothing and try reconnecting again lazily.
To improve the user experience, we should look into standardizing this behavior.
In all cases we should print out an error message that describes the problem, at least saying that the component failed to start. If possible, then, provide more details, and it would be great to add a link to the Dapr docs to point people to the component's specs.
What happens next (crash or continue working without the component) is up for discussion.
Note that some components maintain a persistent connection, so connection failures can be determined in the "init" methods directly (e.g. databases). Other components are stateless (e.g. Azure Storage, S3), so we need to decide what to do with those: do we attempt to make a GET request in the "init" method to validate the connection parameters, or do we lazily wait for first use to report a connection issue? (Or that could be configurable?)
I think we should make this behavior be verified by conformance tests - that is where standard behavior is validated.
Now, when it comes to what should be the standard behavior, I think it should be a combination of all above:
- Component should try to connect on init, optionally with built-in retries. If cannot connect, return error.
- Component can optionally have reconnection logic in the background, in case connection is lost post init (this is validated in certification tests)
- Dapr runtime will log error when a component cannot init.
- Dapr runtime should emit a metric for each component that failed/succeeded to initialize, but metric should include the component type and not the component name (since name will vary per customer setup) - this should be validated in conformance tests.
That makes sense to me.
So, to confirm, we should not crash when a component fails to init. Instead, we'll let the app make API calls that will fail.
I think we should make this behavior be verified by conformance tests - that is where standard behavior is validated.
Now, when it comes to what should be the standard behavior, I think it should be a combination of all above:
- Component should try to connect on init, optionally with built-in retries. If cannot connect, return error.
- Component can optionally have reconnection logic in the background, in case connection is lost post init (this is validated in certification tests)
- Dapr runtime will log error when a component cannot init.
- Dapr runtime should emit a metric for each component that failed/succeeded to initialize, but metric should include the component type and not the component name (since name will vary per customer setup) - this should be validated in conformance tests.
This makes sense to me.
That makes sense to me.
So, to confirm, we should not crash when a component fails to init. Instead, we'll let the app make API calls that will fail.
We have metadata to say if a component is required or optional. See https://docs.dapr.io/operations/components/component-schema/
That makes sense to me.
So, to confirm, we should not crash when a component fails to init. Instead, we'll let the app make API calls that will fail.
No, that's not the desired behavior. If a component cannot connect after all retries were exhausted, we return an error and crash as we do today. Users have the option today to ignore crashing and continuing init with ignoreErrors: https://docs.dapr.io/operations/components/component-schema/#format
