Dovecot doesn't search the index

[zyberzero]

Hey!

Thanks for this project.

I'm having some troubles though, and I can't figure out how to get further.

I've compiled fts-elastic, installed it, and setup dovecot as per instructions.

When I run

doveadm fts rescan -u [email protected]
doveadm index -u user@domain -q '*'

The messages gets indexed.

I can see the messages in the ES index, curl http://localhost:9200/m/_search gives me the most recently indexed messages.

My 90-fts.conf looks like:

# cat /etc/dovecot/conf.d/90-fts.conf 
mail_plugins = $mail_plugins fts fts_elastic

plugin {
    fts = elastic
    fts_elastic = debug url=http://localhost:9200/m/ bulk_size=5000000 refresh=fts rawlog_dir=/var/log/fts-elastic/
    fts_autoindex = yes
    fts_enforced = yes
    fts_autoindex_exclude = \Trash
}

service indexer-worker {
    # Increase vsz_limit to 2GB or above.
    # Or 0 if you have rather large memory usable on your server, which is preferred for performance)
    vsz_limit = 2G
}

However, when doing IMAP searches, I can't see any log messages that inidicates that the request is going to ES.

When enabling mail_debug in dovecot I can see it doing a sequential scan of all the messages on disk. (4 SEARCH (BODY 'test'))

I can't see no new log files in /var/log/fts-elastic when searching, only when indexing.

If I do a search to ES

$ curl -X GET "localhost:9200/m/_search?pretty" -H 'Content-Type: application/json' -d'
{
  "query": {
    "match": {
      "user": {
        "query": "[email protected]"
      }
    }
  }
}
'

I get the proper response, within a second or so.

Do you have an idea why Dovecot won't use this plugin?

ES 7.16, Dovceot 2.3.4.1, fts-elastic d6291f1364a661653823c63e40e05d550edac88e

Thanks!

[filiphanes]

Could you use newest or newer dovecot version?

[zyberzero]

@filiphanes , I've upgraded to the latest dovecot version available in the Debian community package (2.3.17.1 (476cd46418)).

I've recompiled fts-elastic to use the new Dovecot libraries.

Then, if I activate fts-elastic it crashes when selecting an inbox.

Starting program: /usr/lib/dovecot/imap -u [email protected]
* PREAUTH [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS BINARY MOVE SNIPPET=FUZZY PREVIEW=FUZZY PREVIEW STATUS=SIZE SAVEDATE LITERAL+ NOTIFY SPECIAL-USE] Logged in as [email protected]
1 SELECT INBOX

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff7a520ae in ?? () from /usr/lib/dovecot/modules/lib20_fts_plugin.so
(gdb) bt
#0  0x00007ffff7a520ae in ?? () from /usr/lib/dovecot/modules/lib20_fts_plugin.so
#1  0x00007ffff7a5af66 in fts_mailbox_allocated () from /usr/lib/dovecot/modules/lib20_fts_plugin.so
#2  0x00007ffff7ec45f8 in hook_mailbox_allocated () from /usr/lib/dovecot/libdovecot-storage.so.0
#3  0x00007ffff7ebf967 in mailbox_alloc () from /usr/lib/dovecot/libdovecot-storage.so.0
#4  0x000055555556e462 in cmd_select_full ()
#5  0x00005555555762c4 in command_exec ()
#6  0x00005555555743af in ?? ()
#7  0x0000555555574461 in ?? ()
#8  0x0000555555574835 in client_handle_input ()
#9  0x0000555555574d32 in client_input ()
#10 0x00007ffff7daf679 in io_loop_call_io () from /usr/lib/dovecot/libdovecot.so.0
#11 0x00007ffff7db0c61 in io_loop_handler_run_internal () from /usr/lib/dovecot/libdovecot.so.0
#12 0x00007ffff7daf71c in io_loop_handler_run () from /usr/lib/dovecot/libdovecot.so.0
#13 0x00007ffff7daf8a0 in io_loop_run () from /usr/lib/dovecot/libdovecot.so.0
#14 0x00007ffff7d233d3 in master_service_run () from /usr/lib/dovecot/libdovecot.so.0
#15 0x0000555555565c6d in main ()

I've been playing along with another fts-plugin (fts-xapian more precisely) as well, that works though. As soon as I remove fts-elastic from 90-fts.conf it works again.

One thing that I saw was that the build script tries to include -I/usr/lib/dovecot/src/plugins/fts, but I can find that src folder on that location. It builds cleanly however.

Any ideas how to get further?

Thanks!

[filiphanes]

I didn't tested it on this newest version. It might be bug. Next ideas to try:

• double check that new compiled module is used
• use older version, maybe around last winter, because it worked for me and other people
• find and fix the bug

[zeph]

@filiphanes before I'd open a new ticket I spotted this one and it might be related... just blindly guessing here... I noticed that I was having errors in the audit logs related to the indexer and apparmor... I had to forcefully disable it and it started working... to be honest I'm using this package but with the fts_lucene

when I try to attempt a search... the error in the logs is audit: type=1400 audit(1668008674.433:303): apparmor="DENIED" operation="exec" profile="dovecot" name="/usr/lib/dovecot/indexer" pid=39020 comm="dovecot" requested_mask="x" denied_mask="x" fsuid=0 ouid=0

I'm not an apparmor expert here, so... I look forward for your feedback

p.s. I have a different mail_location than the one in the default config

diff --git a/conf.d/10-auth.conf b/conf.d/10-auth.conf
index 1c59eb4..f2d8126 100644
--- a/conf.d/10-auth.conf
+++ b/conf.d/10-auth.conf
@@ -8,6 +8,7 @@
 # connection is considered secure and plaintext authentication is allowed.
 # See also ssl=required setting.
 #disable_plaintext_auth = yes
+disable_plaintext_auth = yes
 
 # Authentication cache size (e.g. 10M). 0 means it's disabled. Note that
 # bsdauth, PAM and vpopmail require cache_key to be set for caching to be used.
diff --git a/conf.d/10-mail.conf b/conf.d/10-mail.conf
index d95112a..41f976b 100644
--- a/conf.d/10-mail.conf
+++ b/conf.d/10-mail.conf
@@ -27,7 +27,9 @@
 #
 # <doc/wiki/MailLocation.txt>
 #
-mail_location = mbox:~/mail:INBOX=/var/mail/%u
+#mail_location = mbox:~/mail:INBOX=/var/mail/%u
+mail_location = mdbox:~/Mail
+## https://wiki2.dovecot.org/MailboxFormat/dbox
 
 # If you need to set multiple mailbox locations or want to change default
 # namespace settings, you can do it by defining namespace sections.
@@ -373,10 +375,14 @@ protocol !indexer-worker {
 
 # Maximum dbox file size until it's rotated.
 #mdbox_rotate_size = 10M
+#zeph#mdbox_rotate_size = 700M
+mdbox_rotate_size = 88M
 
 # Maximum dbox file age until it's rotated. Typically in days. Day begins
 # from midnight, so 1d = today, 2d = yesterday, etc. 0 = check disabled.
 #mdbox_rotate_interval = 0
+mdbox_rotate_interval = 180d
+##mdbox_rotate_interval = 15m
 
 # When creating new mdbox files, immediately preallocate their size to
 # mdbox_rotate_size. This setting currently works only in Linux with some
diff --git a/conf.d/10-master.conf b/conf.d/10-master.conf
index d52ce80..873c929 100644
--- a/conf.d/10-master.conf
+++ b/conf.d/10-master.conf
@@ -16,11 +16,13 @@
 
 service imap-login {
   inet_listener imap {
+    # https://dovecot.org/pipermail/dovecot/2010-March/047479.html
+    port = 0
     #port = 143
   }
   inet_listener imaps {
-    #port = 993
-    #ssl = yes
+    port = 993
+    ssl = yes
   }
 
   # Number of connections to handle before starting a new process. Typically
diff --git a/conf.d/10-ssl.conf b/conf.d/10-ssl.conf
index 134b7b3..1b82c29 100644
--- a/conf.d/10-ssl.conf
+++ b/conf.d/10-ssl.conf
@@ -30,15 +30,10 @@ ssl_key = </etc/dovecot/private/dovecot.key
 # when Dovecot needs to act as an SSL client (e.g. imapc backend or
 # submission service). The directory is usually /etc/ssl/certs in
 # Debian-based systems and the file is /etc/pki/tls/cert.pem in
-# RedHat-based systems. Note that ssl_client_ca_file isn't recommended with
-# large CA bundles, because it leads to excessive memory usage.
-#ssl_client_ca_dir =
+# RedHat-based systems.
 ssl_client_ca_dir = /etc/ssl/certs
 #ssl_client_ca_file =
 
-# Require valid cert when connecting to a remote server
-#ssl_client_require_valid_cert = yes
-
 # Request client to send a certificate. If you also want to require it, set
 # auth_ssl_require_client_cert=yes in auth section.
 #ssl_verify_client_cert = no
diff --git a/conf.d/15-mailboxes.conf b/conf.d/15-mailboxes.conf
index 71076d4..cd5b21b 100644
--- a/conf.d/15-mailboxes.conf
+++ b/conf.d/15-mailboxes.conf
@@ -22,20 +22,18 @@
 #   you want in here, but it's not a good idea to use flags other than the
 #   standard ones specified in the RFC:
 #
-#     \All       - This (virtual) mailbox presents all messages in the
-#                  user's message store.
-#     \Archive   - This mailbox is used to archive messages.
-#     \Drafts    - This mailbox is used to hold draft messages.
-#     \Flagged   - This (virtual) mailbox presents all messages in the
-#                  user's message store marked with the IMAP \Flagged flag.
-#     \Important - This (virtual) mailbox presents all messages in the
-#                  user's message store deemed important to user.
-#     \Junk      - This mailbox is where messages deemed to be junk mail
-#                  are held.
-#     \Sent      - This mailbox is used to hold copies of messages that
-#                  have been sent.
-#     \Trash     - This mailbox is used to hold messages that have been
-#                  deleted.
+#     \All      - This (virtual) mailbox presents all messages in the
+#                 user's message store. 
+#     \Archive  - This mailbox is used to archive messages.
+#     \Drafts   - This mailbox is used to hold draft messages.
+#     \Flagged  - This (virtual) mailbox presents all messages in the
+#                 user's message store marked with the IMAP \Flagged flag.
+#     \Junk     - This mailbox is where messages deemed to be junk mail
+#                 are held.
+#     \Sent     - This mailbox is used to hold copies of messages that
+#                 have been sent.
+#     \Trash    - This mailbox is used to hold messages that have been
+#                 deleted.
 #
 # comment:
 #   Defines a default comment or note associated with the mailbox. This
@@ -77,10 +75,4 @@ namespace inbox {
   #  special_use = \Flagged
   #  comment = All my flagged messages
   #}
-
-  # If you have a virtual "Important" mailbox:
-  #mailbox virtual/Important {
-  #  special_use = \Important
-  #  comment = All my important messages
-  #}
 }
diff --git a/conf.d/20-imap.conf b/conf.d/20-imap.conf
index e60b0cd..f11552f 100644
--- a/conf.d/20-imap.conf
+++ b/conf.d/20-imap.conf
@@ -43,8 +43,7 @@
 
 # ID field names and values to send to clients. Using * as the value makes
 # Dovecot use the default value. The following fields have default values
-# currently: name, version, os, os-version, support-url, support-email,
-# revision.
+# currently: name, version, os, os-version, support-url, support-email.
 #imap_id_send = 
 
 # ID fields sent by client to log. * means everything.
diff --git a/dovecot.conf b/dovecot.conf
index c802011..f25ba00 100644
--- a/dovecot.conf
+++ b/dovecot.conf
@@ -28,6 +28,7 @@
 # If you want to specify non-default ports or anything more complex,
 # edit conf.d/master.conf.
 #listen = *, ::
+listen = 127.0.0.1
 
 # Base directory where to store runtime data.
 #base_dir = /var/run/dovecot/

[zeph]

quick ref... this is the way to disable apparmor just for doing the indexing https://wiki.archlinux.org/title/AppArmor#Disabling_loading