ConduitIO
Entity locking in orchestration layer
Initially, each service (pipeline, connector, processor service) had its own instance lock. The problem was twofold - the services locked the instances only for the duration of the operation inside the service and they did not lock any related instance. Let's take the creation of a processor as an example: it requires us to get the pipeline, lock it in place so it's not modified in the meantime, create the processor, add it to the pipeline, commit the transaction to the DB and only then release the lock.
Now that we have the orchestration layer it should be the responsibility of this layer to lock any entities that will be modified in an operation before making changes. The goal is to make Conduit safe for concurrent use, i.e. multiple requests changing the same resource at the same time.
NB: when a pipeline is running it should probably be locked, when it stops it should be unlocked. Why I'm saying this - it should be kept in mind when designing locks because an unlock can be triggered internally when a pipeline stops because of an error (that won't trigger an orchestrator call).