manticoresearch icon indicating copy to clipboard operation
manticoresearch copied to clipboard

SphinxSE - Misc Errors keep appearing periodically.

Open DaveInTO opened this issue 3 years ago • 1 comments

Describe the bug
Many queries being sent through MariaDB 10 SphinxSE, Intermittently errors are being returned.
Most of the time the query succeeds.
Otherwise.

Error:mysqli::query(): (HY000/1430): There was a problem processing the query on the foreign data source. Data source error: searchd error: index realtime_media_123: syntax error, unexpecte  
Error:mysqli::query(): (HY000/1430): There was a problem processing the query on the foreign data source. Data source error: search query already specified; 's' is redundant  
Error:mysqli::query(): (HY000/1430): There was a problem processing the query on the foreign data source. Data source error: search query already specified; 'f' is redundant  
Error:mysqli::query(): (HY000/1430): There was a problem processing the query on the foreign data source. Data source error: search query already specified; 'n' is redundant  
Error:mysqli::query(): (HY000/1430): There was a problem processing the query on the foreign data source. Data source error: search query already specified; '0' is redundant  
Error:mysqli::query(): (HY000/1430): There was a problem processing the query on the foreign data source. Data source error: search query already specified; ' ' is redundant  
Error:mysqli::query(): (HY000/1430): There was a problem processing the query on the foreign data source. Data source error: search query already specified; 'd' is redundant  
Error:mysqli::query(): (HY000/1430): There was a problem processing the query on the foreign data source. Data source error: search query already specified; ' ' is redundant  
Error:mysqli::query(): (HY000/1430): There was a problem processing the query on the foreign data source. Data source error: search query already specified; ' ' is redundant  
Error:mysqli::query(): (HY000/1430): There was a problem processing the query on the foreign data source. Data source error: search query already specified; 's' is redundant  
Error:mysqli::query(): (HY000/1430): There was a problem processing the query on the foreign data source. Data source error: search query already specified; ' ' is redundant  
Error:mysqli::query(): (HY000/1430): There was a problem processing the query on the foreign data source. Data source error: search query already specified; '3' is redundant  
Error:mysqli::query(): (HY000/1430): There was a problem processing the query on the foreign data source. Data source error: search query already specified; 'h' is redundant  
Error:mysqli::query(): (HY000/1430): There was a problem processing the query on the foreign data source. Data source error: search query already specified; '2' is redundant  
Error:mysqli::query(): (HY000/1430): There was a problem processing the query on the foreign data source. Data source error: searchd error: index realtime_media_123: syntax error, unexpecte  
Error:mysqli::query(): (HY000/1430): There was a problem processing the query on the foreign data source. Data source error: searchd error: index realtime_media_123: syntax error, unexpecte  
Error:mysqli::query(): (HY000/1430): There was a problem processing the query on the foreign data source. Data source error: search query already specified; '8' is redundant  
Error:mysqli::query(): (HY000/1430): There was a problem processing the query on the foreign data source. Data source error: search query already specified; '0' is redundant  
Error:mysqli::query(): (HY000/1430): There was a problem processing the query on the foreign data source. Data source error: search query already specified; 'i' is redundant  
Error:mysqli::query(): (HY000/1430): There was a problem processing the query on the foreign data source. Data source error: searchd error: index realtime_media_123: syntax error, unexpecte  
Error:mysqli::query(): (HY000/1430): There was a problem processing the query on the foreign data source. Data source error: searchd error: index realtime_media_123: syntax error, unexpecte  
Error:mysqli::query(): (HY000/1430): There was a problem processing the query on the foreign data source. Data source error: search query already specified; 't' is redundant  
Error:mysqli::query(): (HY000/1430): There was a problem processing the query on the foreign data source. Data source error: search query already specified; 'e' is redundant  
Error:mysqli::query(): (HY000/1430): There was a problem processing the query on the foreign data source. Data source error: search query already specified; ' ' is redundant  
Error:mysqli::query(): (HY000/1430): There was a problem processing the query on the foreign data source. Data source error: search query already specified; 't' is redundant  
Error:mysqli::query(): (HY000/1430): There was a problem processing the query on the foreign data source. Data source error: searchd error: index realtime_media_123: syntax error, unexpecte  
Error:mysqli::query(): (HY000/1430): There was a problem processing the query on the foreign data source. Data source error: searchd error: index realtime_media_123: syntax error, unexpecte  
Error:mysqli::query(): (HY000/1430): There was a problem processing the query on the foreign data source. Data source error: search query already specified; 'n' is redundant  
Error:mysqli::query(): (HY000/1430): There was a problem processing the query on the foreign data source. Data source error: searchd error: index realtime_media_123: syntax error, unexpecte  
Error:mysqli::query(): (HY000/1430): There was a problem processing the query on the foreign data source. Data source error: searchd error: index realtime_media_123: syntax error, unexpecte  

To Reproduce
Steps to reproduce the behavior:

  1. SphinxSE Storage engine
    Sample queries
 (sphinxmedia.query='mode=extended2;index=realtime_media_123;@indexedTags en;filter=moderationstatus,1;filter=vhost,123;filter=parentgroup,422841;range=channelleft,302,373;filter=status,3;filter=context,1;filter=hidden,0;filter=filetype,1;sort=extended:upload DESC,upload DESC;limit=16;offset=16;maxmatches=1000000;')   

Expected behavior
A clear and concise description of what you expected to happen.

Describe the environment:

- Manticore Search 3.6  
- Linux ec2-db-slave2 5.8.0-1041-aws #43~20.04.1-Ubuntu SMP Thu Jul 15 11:07:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux  
- MariaDB 10.6  

Messages from log files:
Messages from searchd.log and query.log (if applicable).

Additional context

CREATE TABLE `sphinxmedia` (  
  `id` bigint(10) unsigned NOT NULL,  
  `weight` int(11) NOT NULL,  
  `query` varchar(1000) NOT NULL,  
  `_sph_geodist` float DEFAULT NULL,  
  KEY `query` (`query`)  
) ENGINE=SPHINX DEFAULT CHARSET=utf8mb3 CONNECTION='sphinx://172.30.3.35:9312'  

172.30.3.35 is an HaProxy server load balancing the queries
Indextool runs fine on the indexes.
I've tried isolating it to a specific searchd but all of them are doing the same.

Index conf  
  index realtime_media_123  
        {  
                rt_mem_limit = 2048M  
                        type     = rt  
                        dict = keywords  
                        path     =    /var/lib/manticore/data/realtimemedia_123  
                        rt_field = indexedtitle  
                        rt_field = indexedmessage  
                        rt_field = indexedtags  
                        rt_field = indexedfilename  
                        rt_field = indexedfirstname  
                        rt_field = indexedlastname  
                        rt_field = indexeduser  
                        rt_attr_json = indexedmetadata  
                        rt_attr_string = filename  
                        rt_attr_string = title  
                        rt_attr_uint = filetype  
                        rt_attr_uint = status  
                        rt_attr_uint = channel  
                        rt_attr_uint = channelleft  
                        rt_attr_uint = vhost  
                        rt_attr_uint = moderationstatus  
                        rt_attr_uint = context  
                        rt_attr_uint = parentid  
                        rt_attr_uint = parentuserid  
                        rt_attr_uint = uid  
                        rt_attr_uint = privacy  
                        rt_attr_uint = commentcount  
                        rt_attr_uint = approvedcomments  
                        rt_attr_uint = notdeniedcomments  
                        rt_attr_uint = hits  
                        rt_attr_uint = filesize  
                        rt_attr_uint = ofilesize  
                        rt_attr_uint = width  
                        rt_attr_uint = height  
                        rt_attr_uint = length  
                        rt_attr_uint = offensive  
                        rt_attr_uint = votecount  
                        rt_attr_uint = rotation  
                        rt_attr_uint = autoblog  
                        rt_attr_uint = hidden  
                        rt_attr_uint = children  
                        rt_attr_timestamp = upload  
                        rt_attr_timestamp = date  
                        rt_attr_timestamp = converttime  
                        rt_attr_timestamp = lastupdatetime  
                        rt_attr_timestamp = startdate  
                        rt_attr_timestamp = enddate  
                        rt_attr_float = latituderadians  
                        rt_attr_float = longituderadians  
                        rt_attr_float = rating  
                        rt_attr_float = conversiontime  
                        rt_attr_multi = parentchannels  
                        rt_attr_multi = gid  
                        rt_attr_multi = parentgroup  
                        rt_attr_multi = miscsort  
                        min_infix_len = 4  
                        stored_fields = indexedtitle,indexedmessage,indexedtags,indexedfilename,indexedfirstname,indexedlastname,indexeduser  
        }  
searchd  
  
        searchd  
{  
        workers=threads  
                log                     = /var/log/manticore/searchd.log  
                binlog_path             = /var/lib/manticore/binlog  
                query_log               = /var/log/manticore/query.log  
                persistent_connections_limit    = 1  
                network_timeout         = 5  
                client_timeout          = 100  
                rt_flush_period         = 7200  
                binlog_max_log_size = 128M  
                pid_file                = /var/run/manticore/searchd.pid  
                seamless_rotate         = 1  
                preopen_indexes         = 0  
                unlink_old              = 1  
                attr_flush_period       = 900  
                max_packet_size         = 8M  
                max_filters             = 256  
                max_filter_values       = 4096  
                listen_backlog          = 1000  
                listen = 0.0.0.0:9312:sphinx  
                listen = 0.0.0.0:9306:mysql  
                listen = 0.0.0.0:9308:http  
}  

DaveInTO avatar Aug 03 '21 19:08 DaveInTO

Definitely seems related to MariaDB 10.6.. Reverted my servers back to 10.5 and the errors have disappeared

DaveInTO avatar Aug 05 '21 18:08 DaveInTO