liquibase-cassandra icon indicating copy to clipboard operation
liquibase-cassandra copied to clipboard

Lock acquisition concurrency issue

Open denyskonakhevych opened this issue 3 years ago • 2 comments

In order to apply changes correctly, liquibase first need to acquire lock to insure only single active upgrader is now working. This brings us to cases where multiple nodes try to acquire lock simultaneously. Let's look at the case with 2 nodes

1							2

check -> released				check -> released
 ...  							...
acquire						acquire
process		   				process
release 						release
 ...		  		 			...

On the first line, both check is that lock occupied or not with query SELECT COUNT(*) FROM <table_name> where LOCKED = TRUE ALLOW FILTERING

After that both execute following query to acquire lock: UPDATE <table_name> SET LOCKED = TRUE, LOCKEDBY = '<lockedBy>', LOCKGRANTED = <ts> WHERE ID = 1

And result in both cases is -1

I suggest to change query to following UPDATE <table_name> SET LOCKED = TRUE, LOCKEDBY = '<lockedBy>', LOCKGRANTED = <ts> WHERE ID = 1 IF LOCKED = FALSE The result of this operation would still be -1, but record would not be changed because of IF condition

After that we can check whether current host managed to occupy lock: SELECT COUNT(*) FROM <table_name> where LOCKED = TRUE AND LOCKEDBY = '<lockedBy>' ALLOW FILTERING

So, if result is bigger then 0 - current process successfully occupied lock, otherwise - another was faster this time. Now we will see following situation

1							2

check -> released				check -> released
 ...  							...
try acquire					try acquire
check -> acquired				check -> acquired by another process
process		   				wait
release 						...
 ...		  		 			check -> released
 ...		  		 			try acquire
 ...		  		 			check -> acquired
 ...		  		 			...

┆Issue is synchronized with this Jira Bug by Unito

denyskonakhevych avatar Oct 15 '21 10:10 denyskonakhevych

@denyskonakhevych - Thanks for raising this problem. If I understand correctly - the risk of not fixing is that this bug may allow concurrent updates of a schema from two different Liquibase clients. Is that correct?

recampbell avatar May 17 '22 17:05 recampbell

@denyskonakhevych - Thanks for raising this problem. If I understand correctly - the risk of not fixing is that this bug may allow concurrent updates of a schema from two different Liquibase clients. Is that correct?

That's correct. Typically, if you have liquibase as part of start up sequence and now you are redeploying/scaling your app

denyskonakhevych avatar May 17 '22 19:05 denyskonakhevych