Consider implementing a dragonfly stop command

[ashotland]

Doing version upgrade failover tests we encountered the following scenario

Calling shutdown on the master it restarts quickly and the client reconnects to it

We'd like a stop command that will make dragonfly stop accepting request and respond with stopped error to all commands.

Alternatives:

• Develop a way for the controlplane to systemctl stop dragonfly.service
• Stop the host

If we go for this, replica take over should also terminate in stop rather than shutdown.

[adiholden]

After discussing with @ashotland the stop command will stop all open connections and will reject new connections

[chakaz]

I'm stating the obvious here, but there won't be any way to remotely kill the server. I guess that's acceptable.

[chakaz]

Or do you plan to keep the admin port active?

[royjacobson]

Redis has CLIENT PAUSE [WRITE] with established semantics. Do you think they're good enough or do we have a separate use case?

[romange]

I discussed it with @ashotland offline. He is primarily worried about a scenario where a server becomes unresponsive.

1. First, we agreed that it's an unrealistic scenario, because Dragonfly does not have any bugs.
2. Secondly, even if it had a tiny bug, that for an improbable reason would render the server unresponsive, we could not rely on the server to obey this "STOP" command being unresponsive that is. So in any case it's not very useful.

[ashotland]

@romange I think you are mixing between the HA/fail-over scenario and version upgrade (see the description).

This issue is about the later, where we need to proactively stop the master. How do you suggest that we disconnect and prevent connections in this context?

[romange]

What's the final condition for "replica takover" command @royjacobson ? We do close all the connections to maintain consistency. Do we allow them to be reopened again? I would expect that the master reaches a limbo state where it does not allow new connections on the primary port or alternatively send a custom "MOVED" error even for non-cluster mode.

[ashotland]

@romange, According to the design doc the master performs shutdown.

But with our systemd settings clients can reconnect quickly after restart.

We should also consider what to do until replica take over is ready for prime time. Or may have other scenarios where we'll want to stop a master.

@royjacobson - I am not sure that https://redis.io/commands/client-pause/ is suitable as it doesn't seem to close connections or block new connections (but maybe I misunderstand the documentation).

[romange]

I just do not think that STOP is the right tool, for example for systemd restarts: now you need to race with other clients to configure your restarted instance. Also, replica takeover should be ready soon (right @royjacobson ? ) In any case, as Roy said, "CLIENT PAUSE" is the way to do it but I still doubt it covers your cases reliably.

[romange]

Now I read your last sentence. yes, Redis behavior does not cover new connections but we could change these semantics to fit our needs. In any case, I am still not persuaded that a command is the way to go.

[adiholden]

In the MVP and the current implementation of replica takover which is soon to be finished master shuts down. The problem is that after it will shut down the systemd will restart the server. So I believe we should instead shutting down the server just close all connections and not allow new connections, this should be the final step of the replica takeover in master side and not a command to be issued from the control plane, as you said there will be a race between clients that will reconnect to the server when its loading and us calling the stop command. In additions we could add a command that does this logic, but I dont think it should be part of the replica takover flow.

[ashotland]

OK, I am fine with punting on the new command for now and just changing the replica take over end state.

[romange]

per our discussion, closing the issue