Whether there is an automatic stop mechanism for indexer

[yangjinming1062]

Describe the bug I have a index called Demo, and this is the indexer cmd: quickwit service run indexer --indexes Demo. I run the indexer by supervisord. I wish it to run forever, but sometimes I find that indexer has stopped working.

Expected behavior I would like to know the cause of this problem and how to avoid it.

Configuration: Please provide:

1. quickwit --version: 0.2.1
2. The index_config.yaml

---
version: 0 # File format version.

index_id: Demo

doc_mapping:
    field_mappings:
        - name: id
          type: u64
          fast: true
        - name: raw_content
          type: text
          tokenizer: default
          record: position
search_settings:
    default_search_fields: [raw_content]

sources:
    - source_id: source-kafka
      source_type: kafka
      params:
          topic: UserAction
          client_params:
              bootstrap.servers: $(KAFKA)
              group.id: FullText
              security.protocol: PLAINTEXT

3. The quickwit.yaml

# ============================ Quickwit Configuration ==============================

version: 0

listen_address: 0.0.0.0
rest_listen_port: 7280

metastore_uri: file://./qwdata/indices#polling_interval=30s

[fulmicoton]

@yangjinming1062 thank you for the report.

Sources are able to let the Indexer know that there are no more message, and there won't be any more message in the future. In that case, the indexer may shut down gracefully. This is used for the file indexer for instance. For Kafka however, it should run forever.

I remember there used to be a bug in a released version of Quickwit that would cause an indexing pipeline to terminate apparently successfully. It has been fixed since then. Maybe upgrading Quickwit could be a solution.

Another possibility is if Quickwit has hit one of the limit of your system: e.g. OOM Killer, or number of open file descriptor. If you have only one index running, this is unlikely.

Can you share your logs? They should have plenty of information that will help answer your question.

[yangjinming1062]

Thank you for your answer. There's only one index running, and the cmd running in the k8s. I'm sure the pod didn't reboot.Additional information is that Kafka can have a large amount of data.

[yangjinming1062]

Additional information： I just noticed that the indexer in the container has stopped again, so I into the container an ps process When I restart indexer, I see indexer's log in the container as follows, and it looks like it's back to normal.

[evanxg852000]

Can you please try the latest release version of Quickwit, form the log you are using v0.2.1. The latest is v0.3.1

[yangjinming1062]

Can you please try the latest release version of Quickwit, form the log you are using v0.2.1. The latest is v0.3.1

Yes, I will try the latest version and watch to update the status of this problem.

[fmassot]

any news on this @yangjinming1062 ?

[fmassot]

@yangjinming1062 Closing as this should does not happen in the latest version. Feel free to reopen it if you manage to reproduce this issue with 0.5. You may be interested by the helm chart to deploy Quickwit on k8s.