Docker Compose example for Elastic Search

[keunes]

Hello,

With the great help of the NC community I've started to use Docker Compose to run my Nextcloud instance. Now I would like to set up full text search. I already have installed the Full text search and the Full text search - Files apps (Core & Provider). Next step is is setting up the Platform app: Full text search - Elasticsearch Platform.

The Platform app documentation refers to the Elastic Search docs for other than 'native' installation on Debian. The single-node cluster instructions includes SSL set-up and all which isn't really necessary I guess (my whole NC instance is behind a reverse proxy). The multi-node cluster instructions looks rather complex for me (I just started with Docker).

So I was wondering: would it be possible to provide an example Docker Compose set-up with Elastic Search in a separate container in the compose file (though with access to the NC application container, so I can correctly set the settings in the full-text search app)?

[martadinata666]

just add the elastic in your compose

es:
    image: elasticsearch:7.17.1
    environment:
      - discovery.type=single-node
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - xpack.security.enabled=true
    user: 1000:1000
    ulimits:
      memlock:
        soft: -1
        hard: -1
    networks: 
      - nextcloud
    volumes:
      - ./esdata/data:/usr/share/elasticsearch/data
      - ./esdata/plugins:/usr/share/elasticsearch/plugins

- xpack.security.enabled=true can be disable, but i prefer enable eitherway.

• exec into elastic install plugins. bin/elasticsearch-plugin install ingest-attachment
• set password for elastic bin/elasticsearch-setup-passwords interactive -u "http://localhost:9200" or use auto, command: bin/elasticsearch-setup-passwords auto write down the password.

[keunes]

exec into elastic install plugins set password for elastic

Wouldn't that have to be (re)done after each restart of the container?

If so; I believe I have seen ways to execute tasks via docker-compose.yml ('start with' or sth).

I'll have a play with it over the weekend. Thanks!

[martadinata666]

if you mount the plugin path outside like my sample above the plugin will persist, still reinstall every upgrade though. the password thing will persist even after upgrade. 7.17.1 is the latest known work, v8 wont do right now.

[keunes]

Thanks @martadinata666. I just set things up, but when entering the container and trying to set the password, I get sh: bin/elasticsearch-reset-password: No such file or directory

And indeed there is no password reset folder under bin. Any hints?

EDIT

It seems that the command you gave is for elastic 8.1 - for 7.17 it is bin/elasticsearch-setup-passwords auto.

Now wondering what 'Name of your index' should be.

ncocc fulltextsearch:test
 
.Testing your current setup:  
Creating mocked content provider. ok  
Testing mocked provider: get indexable documents. (2 items) ok  
Loading search platform. (Elasticsearch) ok  
Testing search platform. fail 
In StaticNoPingConnectionPool.php line 64:
                                        
  No alive nodes found in your cluster

[martadinata666]

the problem with auto is the elastic will define password random, do you write it down? and need some setup on fts admin settings, the elastic user, password, and the index name. And sorry for the command i provided it for elastic 8.

Notice the http://elastic:changeme@es:9200 -> http://elastic:thepassword@the-elastic-host-name:9200

[keunes]

the problem with auto is the elastic will define password random, do you write it down?

Yep, I had written it down. I just changed the Servlet address and now the test succeeds. Thanks @martadinata666!

I'll keep this open for now as I want to do a PR with example configuration (as was my initial request) :)

[BenRoe]

@keunes can you provide a full example.

[level420]

@martadinata666 how do you start indexing on startup of the docker stack? In my stack I've defined service dependencies to create the elasticsearch container first, then nextcloud. But nextcloud starts much faster then elasticsearch. Currently I'm periodically starting indexing every 5 minutes from outside the stack using a cron job with this command:

/usr/bin/docker exec -u www-data stack-cloud-nextcloud-1 /var/www/html/occ fulltextsearch:live -q

This starts indexing as expected, but after first successfull start, I'm getting the message in the logs:

[fulltextsearch] Warning: Exception while live index: OCA\FullTextSearch\Exceptions\RunnerAlreadyUpException - Index is already running

[martadinata666]

CMIIW, there are two types of indexing. The First is live mode which runs and stays there forever and automatically detects files. The second is oneshot mode /var/www/html/occ fulltextsearch:index -q that triggered using cron.

Technically I don't know the correct name for it, should ask the developer, live mode and oneshot mode but it reflect what it doing

This should help https://github.com/nextcloud/fulltextsearch/wiki/Commands

[level420]

@martadinata666 Thank you for the explanation, but I was maybe not clear enough with my question:

How is indexing started in nextcloud? Is live mode started automatically when nextcloud starts up? And what happens if the indexing provider elasticsearch is not yet ready when nextcloud attempts to start live indexing? Or is live indexing started with the first run of cron.php?

I apologize if I've overseen this in the documentation, but I've searched a reasonable time to find this information, but couldn't find the answer.

[level420]

The goal would be to have live indexing started when having both, nextcloud and elasticsearch running within a docker stack (docker compose) automatically, without having to start fulltextsearch:live from outside the stack.

[martadinata666]

One way to do it is using a custom docker image, referenced https://github.com/nextcloud/docker/blob/master/.examples/dockerfiles/cron/apache/supervisord.conf, you can add startup fulltextsearch:live after some seconds. sleep XXseconds && start fulltextsearch:live. In this method I'm not really sure about the reliability due maybe to race conditions with elastic, There is also startup control in compose like this https://docs.docker.com/compose/compose-file/05-services/#depends_on but it getting more and more complicated. Ultimately ensure elastic instance is started and ready to receive data.

I personally use that oneshot + cron per 15 minutes, if it fails either fail to connect elastic instance due the instance not ready (most of the case) or some other reason (rarely happen but who knows) so be it, it will re-execute next time.

[level420]

@martadinata666 thank you again for your proposals. What I'm currently doing is to execute occ fulltextsearch:live -q every 5 minutes from outside the stack via the hosts cron. That works, but leaves every 5 minutes, after the first successful run, a message in the nextcloud log Exception while live index: OCA\FullTextSearch\Exceptions\RunnerAlreadyUpException - Index is already running. That's odd because it fills the log file with unnecessary messages. Note that I'm using the parameter -q but that does not stop that messages.

[martadinata666]

Again :live command only need executed once and it will stay in background. With cron only need occ fulltextsearch:index.