LOCK/UNLOCK

[sanikolaev]

For backup script we need to implement:

LOCK <idx>  
UNLOCK <idx>  

which would safely lock/unlock the index for doing maintenance with it (backup).

[tomatolog]

I'd also notice that would be a logical lock that prevent write into index (any INSERT\REPLACE\DELETE UPDATE or any global such as optimize or ALTER statements) but allow to search that index locked.

[barryhunter]

How will it work with FLUSH RTINDEX? Would it be outside the lock, or allowed inside?

If want to run the flush before the backup, ideally it should be allowed inside the lock? If only flush before the lock, the is a small chance of changes missed, when backing up the files. There is the binlog, but it not index specific.

Also if the index is locked will concurrent inserts etc be rejected, or queued up in some fashion?

[klirichek]

Locking is now performed on SST stage of replication. All we've going to do - is to make it available for manual intervention aside replication. So, all issues/consequencies are already in game: if your node is selected as donor for replication, the index will be locked, and then released when files transferred.

These very 'lock' and 'unlock' just to be exposed, all the extra things are separate targets to point

[githubmanticore]

➤ Sergey Nikolaev commented:

How will it work with FLUSH RTINDEX? Would it be outside the lock, or allowed inside?

If want to run the flush before the backup, ideally it should be allowed inside the lock? If only flush before the lock, the is a small chance of changes missed, when backing up the files. There is the binlog, but it not index specific.

Good point! We'll need to make sure it's so or perhaps LOCK should do it automatically as part of itself after the lock is acquired.

[barryhunter]

Ah yes, for backup at least a auto-flush would make sense. guess should flush ATTRIBUTES as well as RTINDEX.

Guess, there may be a case for being able to lock (block updates) without actually flushing, but not not sure what would use it for.

Could be LOCK <index> WITH FLUSH; to make it explicit? if an RT index, it does RTINDEX, but always does ATTRIBUTES

[KarelWintersky]

I think, required advenced syntax for LOCK:

LOCK <index> READ;
LOCK <index> WRITE;

Because during backup process other processes can read Index but should not be able to change the Index.

[sanikolaev]

during backup process other processes can read Index but should not be able to change the Index

It will be a default. LOCK ... WRITE may be useful too, but it's out of the scope of this task (lock index to backup it).

[sanikolaev]

This implementation https://github.com/manticoresoftware/manticoresearch/commit/f6fe6408f24251eb1dd65e284c7c9b42593d7792 seems to be a good start, but incomplete, since it doesn't block write operations to a locked index, e.g.:

Session 1:

mysql> create table t(f text);
Query OK, 0 rows affected (0.00 sec)

mysql> lock t;
+-------------------+-----------------------------------+
| file              | normalized                        |
+-------------------+-----------------------------------+
| data/t/t.meta     | /home/snikolaev/data/t/t.meta     |
| data/t/t.ram      | /home/snikolaev/data/t/t.ram      |
| data/t/t.settings | /home/snikolaev/data/t/t.settings |
+-------------------+-----------------------------------+
3 rows in set (0.00 sec)

Session 2:

mysql> select * from t;
Empty set (0.00 sec)

mysql> insert into t values(0,'abc');
Query OK, 1 row affected (0.10 sec)

mysql> select * from t;
+---------------------+------+
| id                  | f    |
+---------------------+------+
| 2811970048334233617 | abc  |
+---------------------+------+
1 row in set (0.42 sec)

Session 1:

mysql> select * from t;
+---------------------+------+
| id                  | f    |
+---------------------+------+
| 2811970048334233617 | abc  |
+---------------------+------+
1 row in set (0.01 sec)

mysql> unlock t;
Query OK, 1 row affected (0.00 sec)

mysql> select * from t;
+---------------------+------+
| id                  | f    |
+---------------------+------+
| 2811970048334233617 | abc  |
+---------------------+------+
1 row in set (0.00 sec)

I.e. while the index was locked I could insert a document with no problem and the document even persisted after I had unlocked the index.

[barryhunter]

As mentioned in the other thread, perhaps worth clarifying is this lock intended for a physical backup or a logical one.

It seems it might prevent writing to the index files, which would enable a physical backup to be taken relatively safely*. Further updates/inserts to index, would just accumulate in memory until next flush. So would be good enough for a clean backup.

But it wouldn't be good enough for a logical backup. Data could be changing even when transactions are used.

• assuming the lock+flush didn't happen in the middle of a transaction - not sure if that is possible.

[sanikolaev]

It seems it might prevent writing to the index files, which would enable a physical backup to be taken relatively safely*.

Right. I misunderstood how it works initially. Actually if we go this way I think we should name it differently: not LOCK/UNLOCK, but perhaps FREEZE/UNFREEZE or smth like that, since it may be too confusing for most more or less experienced users when it's called LOCK: you lock an index, but anyone can still modify the data and can see the modifications. It's confusing since it looks like no locking happened.

This way or another there's still a problem with TRUNCATE:

mysql> lock t;
+-------------------+-----------------------------------+
| file              | normalized                        |
+-------------------+-----------------------------------+
| data/t/t.0.spa    | /home/snikolaev/data/t/t.0.spa    |
| data/t/t.0.spd    | /home/snikolaev/data/t/t.0.spd    |
| data/t/t.0.spds   | /home/snikolaev/data/t/t.0.spds   |
| data/t/t.0.spe    | /home/snikolaev/data/t/t.0.spe    |
| data/t/t.0.sph    | /home/snikolaev/data/t/t.0.sph    |
| data/t/t.0.sphi   | /home/snikolaev/data/t/t.0.sphi   |
| data/t/t.0.spi    | /home/snikolaev/data/t/t.0.spi    |
| data/t/t.0.spm    | /home/snikolaev/data/t/t.0.spm    |
| data/t/t.0.spp    | /home/snikolaev/data/t/t.0.spp    |
| data/t/t.0.spt    | /home/snikolaev/data/t/t.0.spt    |
| data/t/t.meta     | /home/snikolaev/data/t/t.meta     |
| data/t/t.settings | /home/snikolaev/data/t/t.settings |
+-------------------+-----------------------------------+
12 rows in set (0.00 sec)

mysql> truncate rtindex t;
Query OK, 0 rows affected (0.10 sec)

snikolaev@dev:~/data$ ls -la t/
total 16
drwx------ 2 snikolaev snikolaev 4096 Aug  3 09:53 .
drwxrwxrwx 5 syslog    netdev    4096 Aug  3 08:33 ..
-rw------- 1 snikolaev snikolaev    0 Aug  3 08:33 t.lock
-rw------- 1 snikolaev snikolaev  491 Aug  3 09:51 t.meta
-rw------- 1 snikolaev snikolaev    3 Aug  3 09:51 t.settings

i.e. the disk chunks are gone.

Same with DROP, but not only the disk chunks are gone in this case, but all the files except for t.lock:

snikolaev@dev:~/data$ ls -la t/
total 8
drwx------ 2 snikolaev snikolaev 4096 Aug  3 10:02 .
drwxrwxrwx 5 syslog    netdev    4096 Aug  3 10:02 ..
-rw------- 1 snikolaev snikolaev    0 Aug  3 08:33 t.lock

and it also "unlocks" the index automatically.

I think if we:

• fix the truncate/drop issues
• name it differently

it will be ok to use it for physical backups. May be even more beneficial for the user than the real locking functionality as it's normally done, but we then need to make sure the behaviour in the freeze mode is clear for the user, e.g.:

• DROP/TRUNCATE/OPTIMIZE/FLUSH RAMCHUNK and anything else that shouldn't be working when the index is frozen should fail with an error, so an application calling it can react properly
• If INSERT/REPLACE can start failing at some moment (e.g. when rt_mem_limit is exceeded) an error should be returned as well.

Another issue is with binary logging: while the index is locked if I add documents to it, the binary log grows, but once I then stop the searchd the binary log gets just emptied. When the searchd is started back the records appear to be just lost. We should either fix it (so the binlog doesn't get emptied in this case), or each INSERT/REPLACE/DELETE should return a warning saying that the index is frozen and the modified data may be lost.

[githubmanticore]

➤ Sergey Nikolaev commented:

What's left:

• rename LOCK/UNLOCK to FREEZE/UNFREEZE
• update the docs to make it clear FREEZE/UNFREEZE:
  • work differently from what normally happens when you lock smth, i.e. do a physical freeze, not a logical lock
  • are beneficial since you can continue inserting data into a frozen index until some point (which also requires an explanation)
  • don't prevent you from dropping/truncating the index
• solve the delete/replace issue. The MRE is:

mysql> drop table if exists t; create table t(f text); insert into t values(1,'a'); flush ramchunk t; insert into t values(2,'b'); select * from t; lock t; delete from t where id = 2; replace into t values(1,'c'); select * from t; 
-------------- 
drop table if exists t 
-------------- 
 
Query OK, 0 rows affected (0.06 sec) 
 
-------------- 
create table t(f text) 
-------------- 
 
Query OK, 0 rows affected (0.00 sec) 
 
-------------- 
insert into t values(1,'a') 
-------------- 
 
Query OK, 1 row affected (0.01 sec) 
 
-------------- 
flush ramchunk t 
-------------- 
 
Query OK, 0 rows affected (0.08 sec) 
 
-------------- 
insert into t values(2,'b') 
-------------- 
 
Query OK, 1 row affected (0.00 sec) 
 
-------------- 
select * from t 
-------------- 
 
+------+------+ 
| id   | f    | 
+------+------+ 
|    2 | b    | 
|    1 | a    | 
+------+------+ 
2 rows in set (0.00 sec) 
 
-------------- 
lock t 
-------------- 
 
+-----------------------------------+-----------------------------------+ 
| file                              | normalized                        | 
+-----------------------------------+-----------------------------------+ 
| /home/snikolaev/data/t/t.0.spa    | /home/snikolaev/data/t/t.0.spa    | 
| /home/snikolaev/data/t/t.0.spd    | /home/snikolaev/data/t/t.0.spd    | 
| /home/snikolaev/data/t/t.0.spds   | /home/snikolaev/data/t/t.0.spds   | 
| /home/snikolaev/data/t/t.0.spe    | /home/snikolaev/data/t/t.0.spe    | 
| /home/snikolaev/data/t/t.0.sph    | /home/snikolaev/data/t/t.0.sph    | 
| /home/snikolaev/data/t/t.0.sphi   | /home/snikolaev/data/t/t.0.sphi   | 
| /home/snikolaev/data/t/t.0.spi    | /home/snikolaev/data/t/t.0.spi    | 
| /home/snikolaev/data/t/t.0.spm    | /home/snikolaev/data/t/t.0.spm    | 
| /home/snikolaev/data/t/t.0.spp    | /home/snikolaev/data/t/t.0.spp    | 
| /home/snikolaev/data/t/t.0.spt    | /home/snikolaev/data/t/t.0.spt    | 
| /home/snikolaev/data/t/t.meta     | /home/snikolaev/data/t/t.meta     | 
| /home/snikolaev/data/t/t.ram      | /home/snikolaev/data/t/t.ram      | 
| /home/snikolaev/data/t/t.settings | /home/snikolaev/data/t/t.settings | 
+-----------------------------------+-----------------------------------+ 
13 rows in set (0.07 sec) 
 
-------------- 
delete from t where id = 2 
-------------- 
 
Query OK, 1 row affected (0.00 sec) 
 
-------------- 
replace into t values(1,'c') 
-------------- 
 
Query OK, 1 row affected (0.00 sec) 
 
-------------- 
select * from t 
-------------- 
 
+------+------+ 
| id   | f    | 
+------+------+ 
|    1 | c    | 
+------+------+ 
1 row in set (0.00 sec) 
 
# here I do kill -9 <pid> and start the instance back 
 
mysql> select * from t; 
-------------- 
select * from t 
-------------- 
 
ERROR 2013 (HY000): Lost connection to MySQL server during query 
mysql> select * from t; 
-------------- 
select * from t 
-------------- 
 
ERROR 2006 (HY000): MySQL server has gone away 
No connection. Trying to reconnect... 
Connection id:    1 
Current database: *** NONE *** 
 
+------+------+ 
| id   | f    | 
+------+------+ 
|    1 | c    | 
+------+------+ 
1 row in set (0.01 sec) 

As it can be seen delete from t where id = 2 and replace into t values(1,'c') were persisted to the disk after the index was locked.

[sanikolaev]

Done in https://github.com/manticoresoftware/manticoresearch/commit/367738a54527d0c7d47a179918a6bbdfa18fe911