manticoresearch
manticoresearch copied to clipboard
manticoresoftware
table gets invisible after crash
If you run this in 5.0.2 or 5.0.3 3094ae5@220823:
drop table if exists t; CREATE TABLE t ( a int, b text ); REPLACE INTO t (id,a,b) VALUES (1,1,'abc'); SELECT CONCAT(b, TO_STRING(a)) FROM t;
you'll get a crash and the table will become invisible, you won't be able to drop it or create another table with the same name:
mysql> show tables;
Empty set (0.00 sec)
mysql> drop table t;
ERROR 1064 (42000): DROP TABLE failed: unknown index 't'
mysql> create table t(f text);
ERROR 1064 (42000): error adding index 't': directory is not empty: data/t
The config is:
searchd {
listen = 127.0.0.1:9315:mysql
data_dir = data
pid_file = 9315.pid
log = searchd.log
}
The log is:
------- CRASH DUMP END -------
[Mon Aug 29 05:56:02.426 2022] [4140135] watchdog: main process 4140136 crashed via CRASH_EXIT (exit code 2), will be restarted
[Mon Aug 29 05:56:02.426 2022] [4140135] watchdog: main process 4140239 forked ok
[Mon Aug 29 05:56:02.428 2022] [4140239] starting daemon version '5.0.2 348514c@220530 dev (columnar 1.15.4 2fef34e@220522)' ...
[Mon Aug 29 05:56:02.428 2022] [4140239] listening on 127.0.0.1:9315 for mysql
[Mon Aug 29 05:56:02.444 2022] [4140243] binlog: replaying log data/binlog/binlog.001
[Mon Aug 29 05:56:02.444 2022] [4140243] WARNING: binlog: replay error at pos=32
[Mon Aug 29 05:56:02.444 2022] [4140243] binlog: index t: skipped at tid 0 and max binlog tid 0
[Mon Aug 29 05:56:02.444 2022] [4140243] binlog: replay stats: 0 commits; 0 updates, 0 reconfigure; 0 pq-add; 0 pq-delete; 0 pq-add-delete, 1 indexes
[Mon Aug 29 05:56:02.444 2022] [4140243] binlog: finished replaying data/binlog/binlog.001; 0.0 MB in 0.000 sec
[Mon Aug 29 05:56:02.444 2022] [4140243] binlog: finished replaying total 1 in 0.000 sec
[Mon Aug 29 05:56:02.537 2022] [4140239] accepting connections
[Mon Aug 29 05:56:02.537 2022] [4140239] log reopened
[Mon Aug 29 05:56:02.538 2022] [4140246] prereading 0 indexes
[Mon Aug 29 05:56:02.538 2022] [4140246] prereaded 0 indexes in 0.000 sec
The problem has smth to do with the built-in watchdog since with watchdog = 0 the behaviour is different and can be considered OK:
------- CRASH DUMP END -------
[Mon Aug 29 06:02:19.780 2022] [4143150] starting daemon version '5.0.2 348514c@220530 dev (columnar 1.15.4 2fef34e@220522)' ...
[Mon Aug 29 06:02:19.780 2022] [4143150] listening on 127.0.0.1:9315 for mysql
[Mon Aug 29 06:02:19.793 2022] [4143155] WARNING: Relocking t
[Mon Aug 29 06:02:19.793 2022] [4143153] binlog: replaying log data/binlog/binlog.001
[Mon Aug 29 06:02:19.793 2022] [4143153] binlog: index t: recovered from tid 0 to tid 1
[Mon Aug 29 06:02:19.793 2022] [4143153] binlog: replay stats: 1 commits; 0 updates, 0 reconfigure; 0 pq-add; 0 pq-delete; 0 pq-add-delete, 1 indexes
[Mon Aug 29 06:02:19.793 2022] [4143153] binlog: finished replaying data/binlog/binlog.001; 0.0 MB in 0.000 sec
[Mon Aug 29 06:02:19.793 2022] [4143153] binlog: finished replaying total 1 in 0.000 sec
[Mon Aug 29 06:02:19.857 2022] [4143150] accepting connections
[Mon Aug 29 06:02:19.858 2022] [4143157] prereading 1 indexes
[Mon Aug 29 06:02:19.858 2022] [4143157] prereaded 1 indexes in 0.000 sec
...
mysql> show tables;
+-------+------+
| Index | Type |
+-------+------+
| t | rt |
+-------+------+
The problem has smth to do with the built-in watchdog since with watchdog = 0 the behaviour is different and can be considered OK:
------- CRASH DUMP END -------
[Mon Aug 29 06:02:19.780 2022] [4143150] starting daemon version '5.0.2 348514c@220530 dev (columnar 1.15.4 2fef34e@220522)' ...
[Mon Aug 29 06:02:19.780 2022] [4143150] listening on 127.0.0.1:9315 for mysql
[Mon Aug 29 06:02:19.793 2022] [4143155] WARNING: Relocking t
[Mon Aug 29 06:02:19.793 2022] [4143153] binlog: replaying log data/binlog/binlog.001
[Mon Aug 29 06:02:19.793 2022] [4143153] binlog: index t: recovered from tid 0 to tid 1
[Mon Aug 29 06:02:19.793 2022] [4143153] binlog: replay stats: 1 commits; 0 updates, 0 reconfigure; 0 pq-add; 0 pq-delete; 0 pq-add-delete, 1 indexes
[Mon Aug 29 06:02:19.793 2022] [4143153] binlog: finished replaying data/binlog/binlog.001; 0.0 MB in 0.000 sec
[Mon Aug 29 06:02:19.793 2022] [4143153] binlog: finished replaying total 1 in 0.000 sec
[Mon Aug 29 06:02:19.857 2022] [4143150] accepting connections
[Mon Aug 29 06:02:19.858 2022] [4143157] prereading 1 indexes
[Mon Aug 29 06:02:19.858 2022] [4143157] prereaded 1 indexes in 0.000 sec
...
mysql> show tables;
+-------+------+
| Index | Type |
+-------+------+
| t | rt |
+-------+------+
I think I have seen this. It only happened once, and been unable to reproduce, so hadnt reported it yet.
In my case at least there was still a .lock, .meta, and .settings files in the index folder. Soon as manually deleted those files, was able to create the table. A worker instance in the helm chart.
seems issue with CREATE TABLE as this call should ends with config save with all indexes but someway it creates index files but does not save the config that is wrong
➤ Aleksey N. Vinogradov commented:
That is wrong guess.
Create table saves indexes ok, however after a crash we resurrected with original index table inherited from watchdog, which never refreshed.
Fixed in https://github.com/manticoresoftware/manticoresearch/commit/6f42ecdc47bd068bb941b26dc437a8bf1eedb00b
FYI: this issue was about that a table gets invisible after a crash. It's fixed. The related issue about the crash itself is https://github.com/manticoresoftware/manticoresearch/issues/940