opencti Opencti : Fail initialize schema, index already exists

Prerequisites

[✔️ ] I read the Deployment and Setup section of the OpenCTI documentation as well as the Troubleshooting page and didn't find anything relevant to my problem.
[ ✔️ ] I went through old GitHub issues and couldn't find anything relevant
[ ✔️ ] I googled the issue and didn't find anything relevant

Description

Hey Community. So I recently configured opencti on an fresh VM there is no docker vollumes mounted and opencti is configured on that vm for the firest time, Below is the error I am getting.

{"category":"APP","errors":[{"attributes":{"genre":"TECHNICAL","http_status":500},"message":"Fail initialize schema, index already exists, previous initialization fail because you kill the platform before the end of the initialization. Please remove your elastic/opensearch data and restart.","name":"CONFIGURATION_ERROR","stack":"CONFIGURATION_ERROR: Fail initialize schema, index already exists, previous initialization fail because you kill the platform before the end of the initialization. Please remove your elastic/opensearch data and restart.\n at error (/opt/opencti/build/src/config/errors.js:8:10)\n at ConfigurationError (/opt/opencti/build/src/config/errors.js:70:53)\n at initializeSchema (/opt/opencti/build/src/database/engine.js:868:11)\n at processTicksAndRejections (node:internal/process/task_queues:95:5)\n at platformInit (/opt/opencti/build/src/initialization.js:96:7)\n at platformStart (/opt/opencti/build/src/boot.js:17:5)"}],"level":"error","message":"Fail initialize schema, index already exists, previous initialization fail because you kill the platform before the end of the initialization. Please remove your elastic/opensearch data and restart.","timestamp":"2024-03-01T06:11:57.929Z","version":"6.0.0"}

Below is my docker compose

version: '3'
services:
  redis:
    image: redis:7.2.4
    restart: always
    volumes:
      - redisdata:/data
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.12.2
    volumes:
      - esdata:/usr/share/elasticsearch/data
    environment:
      # Comment-out the line below for a cluster of multiple nodes
      - discovery.type=single-node
      # Uncomment the line below below for a cluster of multiple nodes
      # - cluster.name=docker-cluster
      - xpack.ml.enabled=false
      - xpack.security.enabled=false
      - thread_pool.search.queue_size=5000
      - logger.org.elasticsearch.discovery="ERROR"
      - "ES_JAVA_OPTS=-Xms${ELASTIC_MEMORY_SIZE} -Xmx${ELASTIC_MEMORY_SIZE}"
    restart: always
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
  minio:
    image: minio/minio:RELEASE.2024-01-16T16-07-38Z
    volumes:
      - s3data:/data
    ports:
      - "9000:9000"
    environment:
      MINIO_ROOT_USER: ${MINIO_ROOT_USER}
      MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD}    
    command: server /data
    restart: always
  rabbitmq:
    image: rabbitmq:3.12-management
    environment:
      - RABBITMQ_DEFAULT_USER=${RABBITMQ_DEFAULT_USER}
      - RABBITMQ_DEFAULT_PASS=${RABBITMQ_DEFAULT_PASS}
      - RABBITMQ_NODENAME=rabbit01@localhost
    volumes:
      - amqpdata:/var/lib/rabbitmq
    restart: always
  opencti:
    image: opencti/platform:6.0.0
    environment:
      - NODE_OPTIONS=--max-old-space-size=8096
      - APP__PORT=8080
      - APP__BASE_URL=${OPENCTI_BASE_URL}
      - APP__ADMIN__EMAIL=${OPENCTI_ADMIN_EMAIL}
      - APP__ADMIN__PASSWORD=${OPENCTI_ADMIN_PASSWORD}
      - APP__ADMIN__TOKEN=${OPENCTI_ADMIN_TOKEN}
      - APP__APP_LOGS__LOGS_LEVEL=error
      - REDIS__HOSTNAME=redis
      - REDIS__PORT=6379
      - ELASTICSEARCH__URL=http://elasticsearch:9200/
      - MINIO__ENDPOINT=minio
      - MINIO__PORT=9000
      - MINIO__USE_SSL=false
      - MINIO__ACCESS_KEY=${MINIO_ROOT_USER}
      - MINIO__SECRET_KEY=${MINIO_ROOT_PASSWORD}
      - RABBITMQ__HOSTNAME=rabbitmq
      - RABBITMQ__PORT=5672
      - RABBITMQ__PORT_MANAGEMENT=15672
      - RABBITMQ__MANAGEMENT_SSL=false
      - RABBITMQ__USERNAME=${RABBITMQ_DEFAULT_USER}
      - RABBITMQ__PASSWORD=${RABBITMQ_DEFAULT_PASS}
      - SMTP__HOSTNAME=${SMTP_HOSTNAME}
      - SMTP__PORT=25
      - PROVIDERS__LOCAL__STRATEGY=LocalStrategy
    ports:
      - "8080:8080"
    depends_on:
      - redis
      - elasticsearch
      - minio
      - rabbitmq
    restart: always
  worker:
    image: opencti/worker:6.0.0
    environment:
      - OPENCTI_URL=http://opencti:8080/
      - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
      - WORKER_LOG_LEVEL=info
    depends_on:
      - opencti
    deploy:
      mode: replicated
      replicas: 3
    restart: always
  connector-export-file-stix:
    image: opencti/connector-export-file-stix:6.0.0
    environment:
      - OPENCTI_URL=http://opencti:8080/
      - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
      - CONNECTOR_ID=${CONNECTOR_EXPORT_FILE_STIX_ID} # Valid UUIDv4
      - CONNECTOR_TYPE=INTERNAL_EXPORT_FILE
      - CONNECTOR_NAME=ExportFileStix2
      - CONNECTOR_SCOPE=application/json
      - CONNECTOR_CONFIDENCE_LEVEL=15 # From 0 (Unknown) to 100 (Fully trusted)
      - CONNECTOR_LOG_LEVEL=info
    restart: always
    depends_on:
      - opencti
  connector-export-file-csv:
    image: opencti/connector-export-file-csv:6.0.0
    environment:
      - OPENCTI_URL=http://opencti:8080/
      - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
      - CONNECTOR_ID=${CONNECTOR_EXPORT_FILE_CSV_ID} # Valid UUIDv4
      - CONNECTOR_TYPE=INTERNAL_EXPORT_FILE
      - CONNECTOR_NAME=ExportFileCsv
      - CONNECTOR_SCOPE=text/csv
      - CONNECTOR_CONFIDENCE_LEVEL=15 # From 0 (Unknown) to 100 (Fully trusted)
      - CONNECTOR_LOG_LEVEL=info
    restart: always
    depends_on:
      - opencti
  connector-export-file-txt:
    image: opencti/connector-export-file-txt:6.0.0
    environment:
      - OPENCTI_URL=http://opencti:8080/
      - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
      - CONNECTOR_ID=${CONNECTOR_EXPORT_FILE_TXT_ID} # Valid UUIDv4
      - CONNECTOR_TYPE=INTERNAL_EXPORT_FILE
      - CONNECTOR_NAME=ExportFileTxt
      - CONNECTOR_SCOPE=text/plain
      - CONNECTOR_CONFIDENCE_LEVEL=15 # From 0 (Unknown) to 100 (Fully trusted)
      - CONNECTOR_LOG_LEVEL=info
    restart: always
    depends_on:
      - opencti
  connector-import-file-stix:
    image: opencti/connector-import-file-stix:6.0.0
    environment:
      - OPENCTI_URL=http://opencti:8080/
      - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
      - CONNECTOR_ID=${CONNECTOR_IMPORT_FILE_STIX_ID} # Valid UUIDv4
      - CONNECTOR_TYPE=INTERNAL_IMPORT_FILE
      - CONNECTOR_NAME=ImportFileStix
      - CONNECTOR_VALIDATE_BEFORE_IMPORT=true # Validate any bundle before import
      - CONNECTOR_SCOPE=application/json,text/xml
      - CONNECTOR_AUTO=true # Enable/disable auto-import of file
      - CONNECTOR_CONFIDENCE_LEVEL=15 # From 0 (Unknown) to 100 (Fully trusted)
      - CONNECTOR_LOG_LEVEL=info
    restart: always
    depends_on:
      - opencti
  connector-import-document:
    image: opencti/connector-import-document:6.0.0
    environment:
      - OPENCTI_URL=http://opencti:8080/
      - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
      - CONNECTOR_ID=${CONNECTOR_IMPORT_DOCUMENT_ID} # Valid UUIDv4
      - CONNECTOR_TYPE=INTERNAL_IMPORT_FILE
      - CONNECTOR_NAME=ImportDocument
      - CONNECTOR_VALIDATE_BEFORE_IMPORT=true # Validate any bundle before import
      - CONNECTOR_SCOPE=application/pdf,text/plain,text/html
      - CONNECTOR_AUTO=true # Enable/disable auto-import of file
      - CONNECTOR_ONLY_CONTEXTUAL=false # Only extract data related to an entity (a report, a threat actor, etc.)
      - CONNECTOR_CONFIDENCE_LEVEL=15 # From 0 (Unknown) to 100 (Fully trusted)
      - CONNECTOR_LOG_LEVEL=info
      - IMPORT_DOCUMENT_CREATE_INDICATOR=true
    restart: always
    depends_on:
      - opencti

  connector-abuseipdb-ipblacklist:
    image: opencti/connector-abuseipdb-ipblacklist:6.0.0
    environment:
      - OPENCTI_URL=http://opencti:8080/
      - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
      - CONNECTOR_ID=4ec5a11d-dab3-4408-ac42-d727629eb206
      - "CONNECTOR_NAME=AbuseIPDB IP Blacklist"
      - CONNECTOR_SCOPE=abuseipdb
      - CONNECTOR_CONFIDENCE_LEVEL=80 # From 0 (Unknown) to 100 (Fully trusted)
      - CONNECTOR_LOG_LEVEL=error
      - ABUSEIPDB_URL=https://api.abuseipdb.com/api/v2/blacklist
      - ABUSEIPDB_API_KEY=<KEY>
      - ABUSEIPDB_SCORE=100
      - ABUSEIPDB_LIMIT=10000
      - ABUSEIPDB_INTERVAL=2 #Day
    restart: always
    depends_on:
      - opencti

volumes:
  esdata:
  s3data:
  redisdata:
  amqpdata:

Environment

OS : Ubuntu 22.04
OpenCTI version: 6.0.0

Reproducible Steps

Steps to create the smallest reproducible scenario:

Set vm_max_cpu_count to 1048575 using sudo sysctl -w vm.max_map_count=1048575
clone official repo from the site (at this time 6.0.0) git clone https://github.com/OpenCTI-Platform/docker.git
configure .env file

OPENCTI_ADMIN_EMAIL=[[email protected]](mailto:[email protected])
OPENCTI_ADMIN_PASSWORD=changeme
OPENCTI_ADMIN_TOKEN=<Valid uuid 4 was here>
OPENCTI_BASE_URL=http://localhost:8080/
MINIO_ROOT_USER=opencti
MINIO_ROOT_PASSWORD=changeme
RABBITMQ_DEFAULT_USER=opencti
RABBITMQ_DEFAULT_PASS=changeme
CONNECTOR_EXPORT_FILE_STIX_ID=dd817c8b-abae-460a-9ebc-97b1551e70e6
CONNECTOR_EXPORT_FILE_CSV_ID=7ba187fb-fde8-4063-92b5-c3da34060dd7
CONNECTOR_EXPORT_FILE_TXT_ID=ca715d9c-bd64-4351-91db-33a8d728a58b
CONNECTOR_IMPORT_FILE_STIX_ID=72327164-0b35-482b-b5d6-a5a3f76b845f
CONNECTOR_IMPORT_DOCUMENT_ID=c3970f8a-ce4b-4497-a381-20b7256f56f0
SMTP_HOSTNAME=localhost
ELASTIC_MEMORY_SIZE=4G

docker compose up -d

Mar 01 '24 06:03 mayank-joshi-01

I have updated the docker compose and other relevant information, in the original question

Mar 01 '24 09:03 mayank-joshi-01

While I know the docs set it to 4G, I have had issues - I usually set it to 8G - ELASTIC_MEMORY_SIZE=8G

Mar 04 '24 14:03 ParamConstructor

@git-SwitchBlade sorry I'm not myself a dev, so it's not clear whehter or not you have found the support you were looking for on this question. Do you still face an issue?

Mar 22 '24 08:03 nino-filigran

Closing this ticket since the last message before mine was 3 weeks ago. Please, @git-SwitchBlade and @ParamConstructor , if you think I've closed it too soon, or if you have any other remark/comment, feel free to re-open it.

Mar 28 '24 09:03 nino-filigran