Allow for top-level Elasticsearch `mapping` to be defined

[jvanderen1]

It would be nice to define an explicit Elasticsearch mapping (minus any _meta fields` at the top of the schema. Something akin to what is in the Elasticsearch documentation:

PUT /my-index-000001
{
  "mappings": {
    "properties": {
      "age":    { "type": "integer" },  
      "email":  { "type": "keyword"  }, 
      "name":   { "type": "text"  }     
    }
  }
}

[lllleonnnn]

I believe you can do this by setting any mapping you want inside transform.mapping: {} in the schema.json file. We use this to create abstract mappings that apply to some downstream transforms - working well so far.

[jvanderen1]

@nowfred The problem is that we use custom plugins to shape the schema into a custom format before indexing. Our mappings don't necessarily align with PGSync's schema output.

We need a way to define a custom Elasticsearch mappings without referencing columns.

[toluaina]

I have added support for this in the master branch. Please give it a try. Here is a schema example

[jvanderen1]

@toluaina That's exciting! I will give it a shot 👍🏻

[jvanderen1]

@toluaina Unfortunately, this does not seem to work for me. I added the following field:

[
  {
    ...
    "mappings": {
      "properties": {
        "name": {
          "type": "text",
          "fields": {
            "raw": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        }
      }
    }
  }
]

I receive the following error:

elasticsearch.BadRequestError: BadRequestError(400, 'mapper_parsing_exception', 'Root mapping definition has unsupported parameters:  [index : {properties={collection={type=text, fields={raw={ignore_above=256, type=keyword}}}}}]')

[Monstrofil]

Same for me, mappings do not work on elastic 7.x.

root@d1694a84f89a:/usr/share/elasticsearch# curl localhost:9200/
{
  "name" : "elasticsearch",
  "cluster_name" : "main",
  "cluster_uuid" : "8j25yDOJTVyO7MjF-xe1JA",
  "version" : {
    "number" : "7.17.24",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "fcf25fff740db6ab3ed5d145c58d70e4c3528ea7",
    "build_date" : "2024-09-05T07:34:51.812485320Z",
    "build_snapshot" : false,
    "lucene_version" : "8.11.3",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

Most likely it's due to changes in 7.x https://www.elastic.co/guide/en/elasticsearch/reference/6.7/indices-create-index.html#_skipping_types