event-gateway icon indicating copy to clipboard operation
event-gateway copied to clipboard

Fix race condition in configuration

Open mthenw opened this issue 6 years ago • 1 comments

There is a race condition in configuration API that may cause inconsistent state and weird behavior of different instances in the same cluster. It's mainly related to creating subscriptions with parametrized paths. The way that it currently works is that during subscription creation path tree is created to check if the path being added doesn't conflict with existing paths. It doesn't happen in transaction (because there is no support for transactions in libkv) which means that in the meantime some new path can be added that conflicts with path being added but it won't be detected.

Potential solution is to switch to native etcd client and use transactions. But I'm not 100% sure it will solve this problem without changing the key structure.

Code: https://github.com/serverless/event-gateway/blob/master/libkv/subscription.go#L38-L43 Related: https://github.com/serverless/event-gateway/pull/130#pullrequestreview-50618980

mthenw avatar Jul 18 '17 14:07 mthenw

context:

we do certain checks for things when configuring, but they are not correct, because updates may come in that break the condition after we check it, but before we write our own new config. the solution is to use a CAS on a global view of the state for each conf item.

spacejam avatar Jul 18 '17 14:07 spacejam