vitess icon indicating copy to clipboard operation
vitess copied to clipboard

Bug Report: When using MySQL without mysqld_safe a stale lock file prevents restarts

Open dbussink opened this issue 1 year ago • 2 comments

Overview of the Issue

When we run MySQL with mysqlctld but without mysqld_safe and MySQL has an OOM issue, it stops with leaving a lock file in place for the MySQL socket.

In case of mysqlctld though, we know that we manage mysqld and that we can safely restart, making it more robust against issues like OOM in this scenario.

Reproduction Steps

Run mysqld with mysqlctld but without mysqld_safe. In case MySQL OOMs, it won't restart MySQL.

Binary Version

-

Operating System and Environment details

-

Log Fragments

I1119 20:13:57.325598       1 mysqld.go:388] Mysqld.Start(1700424834) stderr: 2023-11-19T20:13:57.325526Z 0 [ERROR] [MY-010119] [Server] Aborting
I1119 20:13:57.325590       1 mysqld.go:388] Mysqld.Start(1700424834) stderr: 2023-11-19T20:13:57.325512Z 0 [ERROR] [MY-010268] [Server] Unable to setup unix socket lock file.
I1119 20:13:57.325549       1 mysqld.go:388] Mysqld.Start(1700424834) stderr: 2023-11-19T20:13:57.325477Z 0 [ERROR] [MY-010259] [Server] Another process with pid 15 is using unix socket file.

dbussink avatar Nov 20 '23 13:11 dbussink

Looks like https://github.com/vitessio/vitess/pull/14553 doesn't not properly fix this, as there can be a newline in the file and it then fails with:

E1124 14:11:50.284750       1 mysqld.go:450] Mysqld.Start(1700835110): error parsing pid from lock file: strconv.Atoi: parsing "162\n": invalid syntax
E1124 14:11:50.284993       1 mysqlctld.go:125] failed to start mysqld: strconv.Atoi: parsing "162\n": invalid syntax

dbussink avatar Nov 24 '23 14:11 dbussink

Reopening this one as there's still cases where this can happen. One theory is that it could be that mysqlctld gets assigned the same PID as in the lock file.

Additionally, we can improve the logging to see more information about the other process.

dbussink avatar Mar 05 '24 19:03 dbussink