Index gets stale

[Bubu]

I'm not exactly sure what I'm doing wrong but I keep running in situations where newer mails seem not searchable unless I explicitly call

doveadm fts rescan -u [email protected]
doveadm index -u user@domain -q '*'

for that user.

Here's my config:

mail_plugins = $mail_plugins fts fts_elastic

plugin {
  fts = elastic
  fts_elastic = url=http://localhost:9200/m/ bulk_size=5000000 refresh=fts

# no indexes new emails when user make search
# yes indexes every email when delivered
fts_autoindex = no
fts_autoindex_exclude = \Junk
fts_autoindex_exclude2 = \Trash
}

Any ideas on how to debug this?

[filiphanes]

Elastic you can set refresh.interval https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html#index-refresh-interval-setting in that time new documents are not searchable, if you have this time set long, it may appear that new email are not indexed

Since you have fts_autoindex = no emails are indexed on imap search command and on this first search they may not be visible, because they are queried right after indexing, you can try to wait and search again after refresh interval

You can try setting:

• fts_autoindex = yes this will index each email right after delivery
• fts_elastic = refresh=index this will refresh elastic right after indexing so new emails should be immediately searchable, hmm, but you have refresh=fts, so it should refresh elastic on each search, maybe it does not.

If new emails are not searchable even after several minutes after delivery or last imap search command then there is probably some bug.

[Bubu]

refresh.interval isn't explicitly set, so I guess that defaults to 1s.

Argh, this is so frustrating to debug, because whenever I want to look at it, it seems to be all working and then a month or so later I'll notice missing emails in my searches again :-/.

I remember changing fts_autoindex = yes to no (together with refresh=fts) in the past because that seemed to work better with actually keeping the index up to date. But this has been a while and I haven't done any recent tests.

(I'll keep the issue updated if I discover anything more related to this. Might be a bit until this reoccurs.)

[filiphanes]

Are missing emails only the latest, or there are some "holes"?

[Bubu]

The issues I've noticed to far have always been that emails after a cutoff time weren't indexed. Like "everything newer than 1 month ago" I usually notice if only after a month because K9-mail does a local search in the most recent 200 emails anyway...