replication-manager icon indicating copy to clipboard operation
replication-manager copied to clipboard

When the slave sets maintenance, there is a problem with the status change of proxysql.

Open peterpan104 opened this issue 1 year ago • 9 comments

This is the state when repman parameters, repman and proxysql are first start:

## repman parameter
proxy-servers-read-on-master = false
proxy-servers-read-on-master-no-slave = true

image image

I set 172.24.44.182 as maintenance, the status of repman and proxysql is as follows:

image image

Then set 172.24.47.188 as maintenance, the status of repman and proxysql is as follows:

image image

Now set 172.24.47.188 as online, the status of repman and proxysql is as follows:

image image

then set 172.24.43.182 as online, the status of repman and proxysql is as follows:

image image

proxy-servers-read-on-master-no-slave seems to have no effect, but proxysql status is normal

Now I trigger the switchover, the status of repman and proxysql is as follows:

image image

I set 172.24.43.182 as maintenance, the status of repman and proxysql is as follows:

image image

Then set 172.24.47.188 maintenance, the status of repman and proxysql is as follows:

image image

You can see that 172.24.43.182 is online in the proxysql status!

Now I restore the status to after switchover:

image image

I set 172.24.47.188 as maintenance, the status of repman and proxysql is as follows:

image image

Then set 172.24.43.183 as maintenance, the status of repman and proxysql is as follows:

image image

At this time, proxysql status is still normal.

Now set 172.24.47.188 as online, the status of repman and proxysql is as follows:

image image

You can see that 172.24.43.182 is online in the proxysql status!

I tried setting proxysql-bootstrap=true, It just allows proxy-servers-read-on-master-no-slave to work, However, the status of proxysql will still be the same as the above test. Can this problem be fixed?

peterpan104 avatar Jul 22 '24 09:07 peterpan104

Hi peterpan104 We have to think on it a lot, the solution was never made to allow to put master in maintenance it is just made to avoid that situation to do that and to to preserve the writer traffic . A maintenance would be done on slave one by one , once all slaves have finish maintenance one is promote as master and the old master now a slave or a broken slave will receive the maintenance and get reintroduce as the master

svaroqui avatar Jul 22 '24 11:07 svaroqui

Sorry please correct I set 172.24.43.182 as maintenance on the first lines as it lead to miss understand of the issue , we need to reproduce as what you are trying to do make sens, still i would never stop all my slaves to put all read traffic on a single node

svaroqui avatar Jul 22 '24 11:07 svaroqui

Is this scripting on your side or is it reproductible manually by using the same steps from the gui ?

svaroqui avatar Jul 22 '24 12:07 svaroqui

@svaroqui Sorry, I described it wrong. The above maintenance is all for slave actions.

Under normal conditions, all slaves will not be maintenance. I am simulating the processing after all slaves are failed. The above tests don't use any monitor scripts. The operation can be reproduced through the gui. I will find that it is because my script has set maintenance, But I found that proxysql's changes were not what I expected.

thanks for your reply.

peterpan104 avatar Jul 23 '24 01:07 peterpan104

Do you enable proxysql bootstrap?

caffeinated92 avatar Jul 31 '24 07:07 caffeinated92

HI @caffeinated92 I didn't enable proxysql bootstrap for the above test, but I also enabled this parameter test and the result was the same. The only difference is that when there is no readable machine, master will be used as readable.

peterpan104 avatar Aug 05 '24 07:08 peterpan104

Added bug fix in v2.3.44 but it will need bootstrap to ON. Since we don't want the script to add master in reader list except with acknowledgement

caffeinated92 avatar Aug 07 '24 10:08 caffeinated92

Hi @caffeinated92 I'm using v2.3.44 and setting bootstrap to ON but the problem is not solved. When I set maintenance on both slaves, one will always show online in proxysql. Can this issue be opened and fixed?

peterpan104 avatar Oct 18 '24 07:10 peterpan104

Hi we are investigating and found some issues in the new logic that maintenance state could differ from proxy to proxy

svaroqui avatar Oct 19 '24 15:10 svaroqui

please reopen if not fixed

svaroqui avatar Jan 09 '25 13:01 svaroqui