manticoresearch Strange behavior of searchd indexer for the first run

Steps to reproduce:

Create an idx folder, use the command mkdir idx
We run the Manticore search daemon process with your configuration file using the command: searchd --config ./manticore.conf. Searchd starts but doesn't detect the indexes.
We run the indexer, it works, but since searchd initially didn't see the indexes, it still doesn't see them now, even though the physical index files are being created.
We run the searchd, use searchd --config ./manticore.conf --stop
We run the searchd. Everything works correctly.

Expected result:

Start the searchd. Everything works correctly.
We run the indexer. It starts correctly.

The indexer is functioning, but it doesn't detect the indexes, even though the physical index files are being created.

Logs:

 
Manticore 6.2.12 dc5144d35@230822 (columnar 2.2.4 5aec342@230822) (secondary 2.2.4 5aec342@230822)
 
Copyright (c) 2001-2016, Andrew Aksyonoff
 
Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
 
Copyright (c) 2017-2023, Manticore Software LTD (https://manticoresearch.com)
 

 
using config file '/Users/pavelshilin/Desktop/WORK/manticore_crash_env/manticore.conf'...
 
indexing table 'idx_table_0'...
 
collected 78384 docs, 3.1 MB
 
creating secondary index
 
creating lookup: 78.3 Kdocs, 100.0% done
 
sorted 0.0 Mhits, 100.0% done
 
total 78384 docs, 3135360 bytes
 
total 0.897 sec, 3494492 bytes/sec, 87362.31 docs/sec
 
indexing table 'idx_table_1'...
 
collected 78385 docs, 3.1 MB
 
creating secondary index
 
creating lookup: 78.3 Kdocs, 100.0% done
 
sorted 0.0 Mhits, 100.0% done
 
total 78385 docs, 3135400 bytes
 
total 0.834 sec, 3759269 bytes/sec, 93981.73 docs/sec
 
indexing table 'idx_table_2'...
 
collected 78385 docs, 3.1 MB
 
creating secondary index
 
creating lookup: 78.3 Kdocs, 100.0% done
 
sorted 0.0 Mhits, 100.0% done
 
total 78385 docs, 3135400 bytes
 
total 0.841 sec, 3725421 bytes/sec, 93135.52 docs/sec
 
total 9 reads, 0.001 sec, 980.6 kb/call avg, 0.1 msec/call avg
 
total 202 writes, 0.012 sec, 238.4 kb/call avg, 0.0 msec/call avg
 
rotating tables: successfully sent SIGHUP to searchd (pid=1831).
 
pavelshilin@Pavels-MacBook-Air manticore_crash_env % searchd --config ./manticore.conf --stop           
 
Manticore 6.2.12 dc5144d35@230822 (columnar 2.2.4 5aec342@230822) (secondary 2.2.4 5aec342@230822)
 
Copyright (c) 2001-2016, Andrew Aksyonoff
 
Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
 
Copyright (c) 2017-2023, Manticore Software LTD (https://manticoresearch.com)
 

 
[22:35.284] [37808] using config file '/Users/pavelshilin/Desktop/WORK/manticore_crash_env/manticore.conf' (1806 chars)...
 
[22:35.285] [37808] stop: successfully sent SIGTERM to pid 1831
 
pavelshilin@Pavels-MacBook-Air manticore_crash_env % searchd --config ./manticore.conf                  
 
Manticore 6.2.12 dc5144d35@230822 (columnar 2.2.4 5aec342@230822) (secondary 2.2.4 5aec342@230822)
 
Copyright (c) 2001-2016, Andrew Aksyonoff
 
Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
 
Copyright (c) 2017-2023, Manticore Software LTD (https://manticoresearch.com)
 

 
[22:40.843] [37832] using config file '/Users/pavelshilin/Desktop/WORK/manticore_crash_env/manticore.conf' (1806 chars)...
 
starting daemon version '6.2.12 dc5144d35@230822 (columnar 2.2.4 5aec342@230822) (secondary 2.2.4 5aec342@230822)' ...
 
listening on 127.0.0.1:9312 for sphinx and http(s)
 
listening on 127.0.0.1:9306 for mysql
 
precaching table 'idx_table_0'
 
rotating table 'idx_table_0': success
 
precaching table 'idx_table_1'
 
rotating table 'idx_table_1': success
 
precaching table 'idx_table_2'
 
rotating table 'idx_table_2': success
 
precaching table 'idx_table_rt_0'
 
precaching table 'idx_table_rt_1'
 
precaching table 'idx_table_rt_2'
 
precached 6 tables in 0.010 sec
 
pavelshilin@Pavels-MacBook-Air manticore_crash_env %```

Sep 14 '23 15:09 githubmanticore

➤ sanikolaev commented:

As discussed on today's dev call here's what we can do:

Check if we can react on HUP differently in this case, so the user doesn't have to do RELOAD INDEXES and just indexer --rotate is enough
If we can't, show WARNING: Please run 'RELOAD TABLES' to make the new table(s) visible

Sep 14 '23 15:09 githubmanticore

➤ PavelShilin89 commented:

Indeed, RELOAD TABLES helped solve the problem. The problem was that I was running searchd and indexer in the wrong order.

Sep 14 '23 15:09 githubmanticore

Comments from @klirichek :

Currently implemented behaviour of reloading

On startup daemon loads the config. In case of she-bang, it starts the script once and collects output. Daemon memorize CRC32, size, creation and modification time.

Later, on HUP signal daemon reloads the config (and re-run it in case of she-bang). Then it compares CRC32, size, creation and modification time with memorized values, and in case config wasn't changed - nothing happens.

That code is original solution, 16 years old (implemented in 2008, commit 022d05c0). It is part of a bigger feature " rotate reloads config, adds new indexes and drops old ones".
in case of she-bang, checking creation and modification time looks confusing, as it actually checks meta-data of a script, not of the content.
Checking operation leads to the issue, when a table is not yet created by the indexer, but daemon already runs. In that case 'hup' signal from the indexer will not help; only touching the config or issuing 'reload indexes' works.
Code for add new indexes and drop old ones was rewritten, and it looks like checking became redundant.

Proposition

Remove this config checking. We immediately achieve these things then:

hup signal will bring newborn table online
indexer with --rotate will bring newborn tables online

m.b. implement 'indexer --reload' - as arbitrary way to just send a 'hup' signal (that is - read pidfile and send signal).
M.b. arbitrary send 'hup' even without '--rotate'. This way daemon will automatically pick up new tables when they're born.

From the doc: 'New tables can be loaded by rotation; however, the regular handling of the HUP signal is to check for new tables only if the configuration has changed since server startup. If the table was already defined in the configuration, the table should be first created by running indexer without rotation and perform the RELOAD TABLES statement instead.'

Changes: indexer can be run with or without rotation, it doesn't matter. In first case it creates 'base set of files' and do nothing. In second, it creates 'new set of files' and send HUP. Anyway, daemon first check if config is the same.

You can also run touch config file now. touch manticore.conf && kill -hup $(pgrep searchd).

manticoresearch
manticoresearch copied to clipboard

Strange behavior of searchd indexer for the first run

Steps to reproduce:

Expected result:

Logs:

Currently implemented behaviour of reloading

Proposition

Related snippets and links

manticoresearch manticoresearch copied to clipboard

Strange behavior of searchd indexer for the first run

Steps to reproduce:

Expected result:

Logs:

Currently implemented behaviour of reloading

Proposition

Related snippets and links

manticoresearch
manticoresearch copied to clipboard