Cortex icon indicating copy to clipboard operation
Cortex copied to clipboard

Cortex 3.1.0 can't create ES index

Open jakubgs opened this issue 5 years ago • 44 comments

Work Environment

Question Answer
OS version (server) Ubuntu 20.04
Cortex version / git hash 3.0.1-1
Package Type Binary
ElasticSearch 7.8.0

Problem Description

I started Cortex with a very standard config based on the provided one in which essentially I just changed the url and index. But when i tried starting it up I got:

[error] o.e.d.DBConfiguration [|] ElasticSearch request failure: POST:/cortex_4/_search?scroll=60000ms
StringEntity({"version":"true","query":{"bool":{"must":[{"term":{"relations":{"value":"worker"}}},{"match_all":{}}]}},"from":0,"sort":[{"_id":{"order":"desc"}}]},Some(application/json))
 => ElasticError(index_not_found_exception,no such index [cortex_4],Some(_na_),Some(cortex_4),None,List(ElasticError(index_not_found_exception,no such index [cortex_4],Some(_na_),Some(cortex_4),None,null,None)),None)
[warn] o.e.d.SearchWithScroll [|] Search error
org.elastic4play.IndexNotFoundException$: null
	at org.elastic4play.IndexNotFoundException$.<clinit>(Errors.scala)
	at org.elastic4play.database.DBConfiguration.$anonfun$execute$2(DBConfiguration.scala:145)
	at scala.concurrent.Future.$anonfun$flatMap$1(Future.scala:307)
	at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:41)
	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
	at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
	at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:91)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:85)
	at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:91)

Which went away after I created the cortex_4 index manually. But why was it not created automatically? Does that not happen? But the installation guide in no way mentions that an index needs to be created.

Possible Solutions

As I said I created the index manually, but:

  1. If it needs to be created manually then it should be documented.
  2. If it's not supposed to be created manually why does it complain that it doesn't exist?

jakubgs avatar Oct 13 '20 09:10 jakubgs

I just found this in administration documentation:

Cortex creates specific index schema (mapping) versions in Elasticsearch. Version numbers are appended to the index base name (the 8th version of the schema uses the index cortex_8 if search.index = cortex).

https://github.com/TheHive-Project/CortexDocs/blob/master/admin/admin-guide.md#database

Which indicates to me that Cortex should crate the index. If it's supposed to create the index then why does it fail to start with:

ElasticError(index_not_found_exception,no such index [cortex_4] ...

jakubgs avatar Oct 14 '20 14:10 jakubgs

I found this file which appears to be running database migrations for Cortex: https://github.com/TheHive-Project/Cortex/blob/master/app/org/thp/cortex/models/Migration.scala Which appears to be using a separate library to manage ElasticSearch: https://github.com/TheHive-Project/elastic4play

jakubgs avatar Oct 15 '20 11:10 jakubgs

I thought this is the file that is supposed to create the database: https://github.com/TheHive-Project/elastic4play/blob/2be170cb/app/org/elastic4play/database/DBCreate.scala But this library appears to be using the following ElasticSearch client: https://github.com/sksamuel/elastic4s Which states in the README that to create an index you use createIndex: https://github.com/sksamuel/elastic4s#create-index And there appears to be one case of its use in elastic4play repo: https://github.com/TheHive-Project/elastic4play/search?q=createIndex But it's actually a separate function defined in elastic4play:

  /**
    * Create a new index. Collect mapping for all attributes of all entities
    *
    * @param models list of all ModelAttributes to used in order to build index mapping
    * @return a future which is completed when index creation is finished
    */
  def createIndex(models: Iterable[ModelAttributes]): Future[Unit] = {

https://github.com/TheHive-Project/elastic4play/blob/2be170cbd298dcd4b1e2fb13f7c3d9d8a38cea15/app/org/elastic4play/database/DBIndex.scala#L33-L39 But none of this code actually calls logger to log any info about successful creation of the index. So I have absolutely no clue if this is actually being executed in my case...

jakubgs avatar Oct 15 '20 12:10 jakubgs

Actually, there is something:

      case false if db.version == 1 =>
        logger.info("Create a new empty database")
        Future.successful(0 -> EmptyState)

https://github.com/TheHive-Project/elastic4play/blob/2be170cbd/app/org/elastic4play/services/MigrationSrv.scala#L96-L98

Which suggests that I should see Create a new empty database in my logs somewhere at startup, and it does:

[info] o.e.s.MigrationSrv [|] Create a new empty database
[info] o.e.s.MigrationSrv [|] Migrate database from version 0, add operations for version 2
[info] o.e.s.MigrationSrv [|] Migrate database from version 0, add operations for version 3
[info] o.e.s.MigrationSrv [|] Migrate database from version 0, add operations for version 4
[error] o.e.d.DBConfiguration [|] ElasticSearch request failure: PUT:/cortex_4?
StringEntity({"settings":{"index":{"number_of_shards":3,"number_of_replicas":1,"mapping.nested_fields.limit":100}},"mappings":{"doc":{"date_detection":false,"numeric_detection":false,"properties":{"sequenceCounter":{"type":"long"},"updatedAt":{"t>
 => ElasticError(mapper_parsing_exception,Failed to parse mapping [_doc]: Root mapping definition has unsupported parameters:  [doc : {numeric_detection=false, date_detection=false, properties={data={type=binary}, sequenceCounter={type=long}, end>
[error] o.e.s.MigrationSrv [|] Migration fail
org.elastic4play.InternalError: Unknown error: ElasticError(mapper_parsing_exception,Failed to parse mapping [_doc]: Root mapping definition has unsupported parameters:  [doc : {numeric_detection=false, date_detection=false, properties={data={typ>
        at org.elastic4play.database.DBConfiguration.$anonfun$execute$2(DBConfiguration.scala:148)
        at scala.concurrent.Future.$anonfun$flatMap$1(Future.scala:307)
        at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:41)
        at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
        at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
        at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:91)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:85)
        at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:91)
        at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
[info] o.t.c.s.ErrorHandler [|] POST /api/maintenance/migrate returned 500
org.elastic4play.InternalError: Unknown error: ElasticError(mapper_parsing_exception,Failed to parse mapping [_doc]: Root mapping definition has unsupported parameters:  [doc : {numeric_detection=false, date_detection=false, properties={data={typ>
        at org.elastic4play.database.DBConfiguration.$anonfun$execute$2(DBConfiguration.scala:148)
        at scala.concurrent.Future.$anonfun$flatMap$1(Future.scala:307)
        at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:41)
        at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
        at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
        at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:91)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:85)
        at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:91)
        at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
[error] o.e.d.DBConfiguration [|] ElasticSearch request failure: POST:/cortex_4/_search?
StringEntity({"query":{"match":{"relations":{"query":"user"}}},"size":0},Some(application/json))
 => ElasticError(index_not_found_exception,no such index [cortex_4],Some(_na_),Some(cortex_4),None,List(ElasticError(index_not_found_exception,no such index [cortex_4],Some(_na_),Some(cortex_4),None,null,None)),None)

So it appears it does try to, and then tries to apply migrations, but it fails with:

ElasticError(
  mapper_parsing_exception,
  Failed to parse mapping [_doc]:
    Root mapping definition has unsupported parameters: [
      doc : {
        numeric_detection=false,
        date_detection=false,
        properties={data={type=binary},
        sequenceCounter={type=long},
      end>

But the index is never created because I cannot find it in the cluster:

 > curl -s localhost:9200/_cat/indices
green open .elastichq OtyN7XiTQDa2yAi1h6Awcw 1 1 1 0 13kb 6.5kb

So it tries to apply migrations despite index not existing? Or does it delete it after failed migration attempts?

It's possible that it fails to migrate and then deletes it due to unsupported parameters.

jakubgs avatar Oct 15 '20 12:10 jakubgs

Oooooooh, it appears my ES cluster is too new... I need Elasticsearch 5.6.

https://github.com/TheHive-Project/CortexDocs/blob/master/installation/install-guide.md#1-pre-requisites

You do know that ES 5.6.x has it's End-of-Life on 2019-03-11? That's more than a year ago.

jakubgs avatar Oct 15 '20 12:10 jakubgs

I upgraded our instance to it but it doesn't seem like it's trying to create the index:

[error] o.e.d.DBConfiguration [|] ElasticSearch request failure: POST:/cortex_4/_search?scroll=60000ms
StringEntity({"version":"true","query":{"bool":{"must":[{"term":{"relations":{"value":"worker"}}},{"match_all":{}}]}},"from":0,"sort":[{"_doc":{"order":"desc"}}]},Some(application/json))
 => ElasticError(index_not_found_exception,no such index [cortex_4],Some(_na_),Some(cortex_4),None,List(ElasticError(index_not_found_exception,no such index [cortex_4],Some(_na_),Some(cortex_4),None,null,None,None,None,null)),None,None,None,null)
[warn] o.e.d.SearchWithScroll [|] Search error
org.elastic4play.IndexNotFoundException$: null
        at org.elastic4play.IndexNotFoundException$.<clinit>(Errors.scala)
        at org.elastic4play.database.DBConfiguration.$anonfun$execute$2(DBConfiguration.scala:147)
        at scala.concurrent.Future.$anonfun$flatMap$1(Future.scala:307)
        at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:41)
        at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
        at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:56)
        at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:93)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:85)
        at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:93)

jakubgs avatar Oct 15 '20 12:10 jakubgs

It seems like it's somehow storing state which causes it not to attempt to migrate again.

jakubgs avatar Oct 15 '20 12:10 jakubgs

Based on the error:

[ERROR] from org.elastic4play.database.DBConfiguration in application-akka.actor.default-dispatcher-7
  - ElasticSearch request failure: POST:/cortex_4/_search?scroll=60000ms
    StringEntity(
      {
        "version":"true",
        "query":{
          "bool":{
            "must":[
              {"term":{"relations":{"value":"job"}}},
              {"term":{"status":{"value":"Waiting"}}}
            ]
          }
        },
        "from":0,
        "sort":[
          {"_doc":{"order":"desc"}}
        ]
      },
      Some(application/json)
    )
   => ElasticError(
        index_not_found_exception,
        no such index [cortex_4],Some(_na_),
        Some(cortex_4),
        None,
        List(ElasticError(index_not_found_exception,no such index [cortex_4],Some(_na_),Some(cortex_4),
        None,null,None,None,None,null)),None,None,None,null
    )

Something goes in DBConfiguration module: https://github.com/TheHive-Project/elastic4play/blob/1.12.1/app/org/elastic4play/database/DBConfiguration.scala Which I would assume should run AFTER the database is created, which should be done in: https://github.com/TheHive-Project/elastic4play/blob/1.12.1/app/org/elastic4play/services/MigrationSrv.scala#L96-L98 But clearly isn't being done. No idea why.

jakubgs avatar Oct 22 '20 12:10 jakubgs

It seems to me like Cortex somehow remembers it was already started before and thinks that the DB already exists and hence doesn't try to create it. But as far as I can tell I've removed every single file Cortex uses before doing a fresh installation and configuration to be sure it's starting with a fresh state. And yet it fails to create the ES index.

jakubgs avatar Oct 22 '20 13:10 jakubgs

I created the cortex_4 index by hand and now the error doesn't show up:

[info] o.t.c.s.DockerJobRunnerSrv - Docker is available:
Info{architecture=x86_64, clusterStore=, cgroupDriver=cgroupfs, containers=0, containersRunning=0, containersStopped=0, containersPaused=0, cpuCfsPeriod=true, cpuCfsQuota=true, debug=false, dockerRootDir=/var/lib/docker/100000.100000, storageDriver=overlay2, driverStatus=[[Backing Filesystem, extfs], [Supports d_type, true], [Native Overlay Diff, true]], executionDriver=null, experimentalBuild=false, httpProxy=, httpsProxy=, id=KNBK:S3L6:YYAG:KF54:24FK:543D:YLHM:DQAW:IBM6:TSXO:N5ZD:L77P, ipv4Forwarding=true, images=1, indexServerAddress=https://index.docker.io/v1/, initPath=null, initSha1=null, kernelMemory=true, kernelVersion=5.4.0-1025-gcp, labels=[], memTotal=7819628544, memoryLimit=true, cpus=2, eventsListener=0, fileDescriptors=22, goroutines=36, name=master-01.gc-us-central1-a.thehive.prod, noProxy=, oomKillDisable=true, operatingSystem=Ubuntu 20.04.1 LTS, osType=linux, plugins=Plugins{volumes=[local], networks=[bridge, host, ipvlan, macvlan, null, overlay]}, registryConfig=RegistryConfig{indexConfigs={docker.io=IndexConfig{name=docker.io, mirrors=[], secure=true, official=true}}, insecureRegistryCidrs=[127.0.0.0/8]}, serverVersion=19.03.13, swapLimit=false, swarm=SwarmInfo{cluster=null, controlAvailable=false, error=, localNodeState=inactive, nodeAddr=, nodeId=, nodes=null, managers=null, remoteManagers=null}, systemStatus=[], systemTime=Thu Oct 22 15:53:41 UTC 2020}
[info] o.t.c.s.JobRunnerSrv - The package cortexutils for python has valid version: 2.0.0
[warn] o.t.c.s.JobRunnerSrv - The package cortexutils for python2 hasn't been found
[info] o.t.c.s.JobRunnerSrv - The package cortexutils for python3 has valid version: 2.0.0
[info] play.api.Play - Application started (Prod) (no global state)
[info] p.c.s.AkkaHttpServer - Enabling HTTP/2 on Akka HTTP server...
[info] p.c.s.AkkaHttpServer - Listening for HTTP on /[0:0:0:0:0:0:0:0]:9001

But considering the documentation for Cortex never actually states that you need to create the index yourself, I'd say this is a problem with the either the docs, or the code. Truth be told I'd expect the index to be created at startup, since it's simple, and removes one more step from installation instructions.

jakubgs avatar Oct 22 '20 15:10 jakubgs

Tough now I can't access the page:

 % curl -s localhost:9001/index.html 
A client error occurred on GET /index.html : Resource not found by Assets controller

And the index in ES cluster is empty, no documents at all. Which would suggest to me that no migrations were run.

I'm so confused. Am I supposed to create the index or not?

jakubgs avatar Oct 22 '20 19:10 jakubgs

You must not create the index in Elasticsearch. When the application doesn't find the index, it returns a HTTP error with status 520. This status code is interpreted by the frontend and redirect to the migration page that ask the user to update/create the index.

Which package do you use ? Where did you download it ? Your last error means that the frontend files is not present in your package.

To-om avatar Oct 30 '20 12:10 To-om

Last thing I tried was 3.1.0-RC1 but it doesn't work because it fails to create the index. I was told this should be fixed in 3.1.0.

jakubgs avatar Oct 30 '20 14:10 jakubgs

Cortex 3.1.x requires Elasticsearch 7. Cortex 3.0.x requires Elasticsearch 6. Breaking changes in Elasticsearch doesn't permit us to have an application compatible with the both versions.

To-om avatar Oct 30 '20 15:10 To-om

Yes. I'm aware, but RC1 still is incapable of creating the index in ES 7.

jakubgs avatar Oct 30 '20 18:10 jakubgs

I have installed 3.1.0 release of Cortex, and yet it still fails to create the index:

[error] o.e.d.DBConfiguration - ElasticSearch request failure: POST:/cortex_5/_search?scroll=60000ms
StringEntity({"seq_no_primary_term":"true","query":{"bool":{"must":[{"term":{"relations":{"value":"worker"}}},{"match_all":{}}]}},"from":0,"sort":[{"_doc":{"order":"desc"}}]},Some(application/json))
 => ElasticError(index_not_found_exception,no such index [cortex_5],Some(_na_),Some(cortex_5),None,List(ElasticError(index_not_found_exception,no such index [cortex_5],Some(_na_),Some(cortex_5),None,null,None,None,None,List())),None,None,None,List())
[warn] o.e.d.SearchWithScroll - Search error
org.elastic4play.IndexNotFoundException$: null

Full log: https://gist.github.com/jakubgs/d75327c13f8da0a5c9ce4a126185146b

I'm not sure what I'm supposed to do if it just fail to create the index.

jakubgs avatar Nov 04 '20 15:11 jakubgs

I directly started out with Cortex 3.1.0 and ES 7.9 and had the same chain of problems but managed to workaround it.

Instead of relying on the 520 http error to trigger the maintenance mode, I went to http://127.0.0.1:9001/index.html#!/maintenance, clicked on the migrate button and afterwards Cortex created the necessary index and inital user.

As far as I can tell, the user used to connect to ES needs the cluster monitor permission and sufficient index permissions on your corext index name.

Neuro-HSOC avatar Nov 11 '20 17:11 Neuro-HSOC

As far as I can tell, the user used to connect to ES needs the cluster monitor permission and sufficient index permissions on your corext index name.

But I'm not using any authentication for my ES cluster.

But thanks for showing how to trigger a migration, wasn't aware of this path. Will test later.

jakubgs avatar Nov 12 '20 12:11 jakubgs

Okay, I opened the site available at localhost:9001 to see the http://127.0.0.1:9001/index.html#!/maintenance page which greeted me with an Update Database button: cortex_update_database Which when clicked makes a POST request to http://localhost:9001/api/maintenance/migrate: cortex_post_migrate Which results in the index being created:

 > curl -s localhost:9200/_cat/indices     
green open cortex_5   b62BDfKkTOSqiFh_yidepg 3 1 1 0 6.4kb  624b
green open .elastichq OtyN7XiTQDa2yAi1h6Awcw 1 1 1 0  13kb 6.5kb

I cannot for the life of me comprehend why this software doesn't run the migration by itself at startup, but it doesn't.

jakubgs avatar Nov 13 '20 12:11 jakubgs

And as far as I can tell the installation instructions do not mention the need to click this button or call that API route: https://github.com/TheHive-Project/CortexDocs/blob/master/installation/install-guide.md

What the hell? If the logs are spewing errors about lack of index in the database why should I think that I need to navigate to the site to run the migrations that trigger the index creation? Why does it not happen automatically at service startup when the index doesn't exist? I don't get it.

jakubgs avatar Nov 13 '20 13:11 jakubgs

And of course the API Guide does not mention the /api/maintenance/migrate route in the lightest: https://github.com/TheHive-Project/CortexDocs/blob/master/api/api-guide.md

jakubgs avatar Nov 13 '20 13:11 jakubgs

This doc mentions the page but not the API route: https://github.com/TheHive-Project/CortexDocs/blob/master/admin/updating.md

But only in the context of updating, not actually starting up the serivice.

jakubgs avatar Nov 13 '20 13:11 jakubgs

I created the cortex_4 index by hand and now the error doesn't show up:

[info] o.t.c.s.DockerJobRunnerSrv - Docker is available:
Info{architecture=x86_64, clusterStore=, cgroupDriver=cgroupfs, containers=0, containersRunning=0, containersStopped=0, containersPaused=0, cpuCfsPeriod=true, cpuCfsQuota=true, debug=false, dockerRootDir=/var/lib/docker/100000.100000, storageDriver=overlay2, driverStatus=[[Backing Filesystem, extfs], [Supports d_type, true], [Native Overlay Diff, true]], executionDriver=null, experimentalBuild=false, httpProxy=, httpsProxy=, id=KNBK:S3L6:YYAG:KF54:24FK:543D:YLHM:DQAW:IBM6:TSXO:N5ZD:L77P, ipv4Forwarding=true, images=1, indexServerAddress=https://index.docker.io/v1/, initPath=null, initSha1=null, kernelMemory=true, kernelVersion=5.4.0-1025-gcp, labels=[], memTotal=7819628544, memoryLimit=true, cpus=2, eventsListener=0, fileDescriptors=22, goroutines=36, name=master-01.gc-us-central1-a.thehive.prod, noProxy=, oomKillDisable=true, operatingSystem=Ubuntu 20.04.1 LTS, osType=linux, plugins=Plugins{volumes=[local], networks=[bridge, host, ipvlan, macvlan, null, overlay]}, registryConfig=RegistryConfig{indexConfigs={docker.io=IndexConfig{name=docker.io, mirrors=[], secure=true, official=true}}, insecureRegistryCidrs=[127.0.0.0/8]}, serverVersion=19.03.13, swapLimit=false, swarm=SwarmInfo{cluster=null, controlAvailable=false, error=, localNodeState=inactive, nodeAddr=, nodeId=, nodes=null, managers=null, remoteManagers=null}, systemStatus=[], systemTime=Thu Oct 22 15:53:41 UTC 2020}
[info] o.t.c.s.JobRunnerSrv - The package cortexutils for python has valid version: 2.0.0
[warn] o.t.c.s.JobRunnerSrv - The package cortexutils for python2 hasn't been found
[info] o.t.c.s.JobRunnerSrv - The package cortexutils for python3 has valid version: 2.0.0
[info] play.api.Play - Application started (Prod) (no global state)
[info] p.c.s.AkkaHttpServer - Enabling HTTP/2 on Akka HTTP server...
[info] p.c.s.AkkaHttpServer - Listening for HTTP on /[0:0:0:0:0:0:0:0]:9001

But considering the documentation for Cortex never actually states that you need to create the index yourself, I'd say this is a problem with the either the docs, or the code. Truth be told I'd expect the index to be created at startup, since it's simple, and removes one more step from installation instructions.

How did you create the index?

imad-testing avatar Dec 01 '20 08:12 imad-testing

As my comment said - "by hand" - also known as "manually". But that's old and not relevant anymore.

Read https://github.com/TheHive-Project/Cortex/issues/305#issuecomment-726748902 and https://github.com/TheHive-Project/Cortex/issues/305#issuecomment-726753287.

jakubgs avatar Dec 01 '20 16:12 jakubgs

I have the same issue with a brand new Cortex 3.1.0 and Elasticsearch 7.10.2 installation, Cortex fails to create index cortex_4 when the migrations are run.

logs: https://pastebin.com/LCiprJ1y

nekinie avatar Jan 18 '21 10:01 nekinie

So Cortex is trying to execute the following:

  "settings": {
    "index": {
      "number_of_shards": 5,
      "number_of_replicas": 0,
      "mapping.nested_fields.limit": 100
    }
  },
  "mappings": {
    "doc": {
      "date_detection": false,
      "numeric_detection": false,
      "properties": {
        "sequenceCounter": {
          "type": "long"
        },
        "updatedAt": {
          "type": "date",
          "format": "epo```
          
Which is creating a mapping called doc, mapping names was removed in Elasticseatch 7.

https://www.elastic.co/guide/en/elasticsearch/reference/6.8/mapping.html
https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html

nekinie avatar Jan 18 '21 11:01 nekinie

Looks like mapping is deprecated, I think this is causing the above problem: https://github.com/sksamuel/elastic4s/blob/master/elastic4s-core/src/main/scala/com/sksamuel/elastic4s/requests/indexes/CreateIndexApi.scala#L30

Mapping is being called by elastic4play, I'm surprised this has not caused trouble in thehive?

Edit: The above statement is false

nekinie avatar Jan 18 '21 15:01 nekinie

Hi, i run a new Cortex + Elastic 7 Installation and get the same Error as described, However my Cortex Service crashes without me having the chance of going to Port 9001 and click the Database Update Button. Is there any Solution to this Issue?.

JustCodeIt4Head avatar Feb 06 '21 23:02 JustCodeIt4Head

Same here

yugoslavskiy avatar Feb 08 '21 23:02 yugoslavskiy

I have the same issue with a brand new Cortex 3.1.0 and Elasticsearch 7.10.2 installation, Cortex fails to create index cortex_4 when the migrations are run.

logs: https://pastebin.com/LCiprJ1y

@nekinie Cortex 3.1.0 should create a cortex_5 index and not cortex_4. Can you confirm you are not using Cortex 3.0.1 with ES7 ?

jeromeleonard avatar Feb 09 '21 19:02 jeromeleonard