elasticsearch icon indicating copy to clipboard operation
elasticsearch copied to clipboard

Published 20 hours ago • verified publisher icon elastic

semantic_text as string type in ES|QL - support for functions and operators

Open ioanatia opened this issue 1 year ago • 3 comments

tracked in https://github.com/elastic/elasticsearch/issues/115103 Related: https://github.com/elastic/elasticsearch/pull/114334 that disallows functions to return TEXT type values. We need to pay attention to this one, it will likely be merged first.

Description

This adds support for semantic_text as a string type in ES|QL and checks the support for functions and operators (https://www.elastic.co/guide/en/elasticsearch/reference/master/esql-functions-operators.html).

The process was:

  • make semantic_text a string type: https://github.com/elastic/elasticsearch/blob/167e55bc63e559fa6f5e05584a2d215a50034955/x-pack/plugin/esql-core/src/main/java/org/elasticsearch/xpack/esql/core/type/DataType.java#L372-L377
  • run the tests and fix any failures
  • for each function/operator that supports keyword/text (docs)write CSV tests to use them with semantic_text; fix any failures
  • for each function/operator that support semantic_text ensure we have test coverage when the arguments are of type semantic_text

Next

This PR takes care of only functions/operator support. Things that are not covered by tests here, but will be in a follow up:

  • the STATS command:
    • aggregate functions for STATS: docs
    • grouping functions for STATS e.g. BUCKET docs
  • support for all commands (from docs) e.g.:
    • STATS
    • DISSECT
    • ENRICH
    • GROK
    • LOOKUP
    • INLINESTATS

If preferred I can make the rest of the changes here to ensure the support for commands too, but I figured it might be easier to review this in 2 parts.

ioanatia avatar Oct 21 '24 14:10 ioanatia

@elasticmachine update branch

ioanatia avatar Oct 22 '24 15:10 ioanatia

Pinging @elastic/es-analytical-engine (Team:Analytics)

elasticsearchmachine avatar Oct 22 '24 15:10 elasticsearchmachine

@craigtaverner do you have time to take a look at this one?

astefan avatar Oct 30 '24 15:10 astefan

@elasticmachine update branch

ioanatia avatar Oct 31 '24 19:10 ioanatia

💔 Backport failed

The backport operation could not be completed due to the following error:

An unexpected error occurred when attempting to backport this PR.

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 115243

elasticsearchmachine avatar Nov 01 '24 06:11 elasticsearchmachine