Peng Huo

Add metadatalog interface and default in memory implementation

1

### Changes 1. Add metadatalog interface which is used by stream source, microbatch execution. 2. Add default in memory implementation of metadatalog. ### Check List - [x] New functionality includes...

_**Stream source interface**_ * **Optional getLatestOffset()** * return the latest **Offset** which map to S3Metadata. **Notice**, the StreamSource does NOT guarantee the Offset is mapping to **unread files**. * return...

## Requirements 1. Add Filesystem storage engine implementation. May leverage hadoop filesystem SDK. 3. Support Local File system. 4. Support AWS S3

Add StreamPlan and MicroBatchExecution

1

1. StreamPlan is a type of AbstractPlan which execute in continuesely 2. MicroBatchExecution is one type of stream execution mode. ## Limitations 1. Currently, we did not consider coordination node...

Disable doctest for 3.0 build

3

## issues Disable doctest because ml-commons snapshot is not ready yet. ## next step enable doctest when https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/3.0.0/latest/linux/x64/tar/builds/opensearch/plugins/opensearch-ml-3.0.0.0.zip is ready

Better document limitation on Array support.

4

**Is your feature request related to a problem? Please describe.** Document current limitation on Array support. https://github.com/opensearch-project/sql/blob/main/opensearch/src/main/java/org/opensearch/sql/opensearch/data/utils/OpenSearchJsonContent.java#L132

**What is the bug?** Error message is misleading ``` "{\n \"status\": 404,\n \"error\": {\n \"type\": \"IndexNotFoundException\",\n \"reason\": \"There was internal problem at backend\",\n \"details\": \"no such index [flint_glue_table_skipping_index]\"\n }\n}" ```...

[BUG] Async Job query can longer than 10280 chars

3

``` POST _plugins/_async_query {"query":"CREATE MATERIALIZED VIEW\n mys3.default.aws_elb_mview AS\n SELECT\n type as `aws.elb.elb_type`,\n time as `@timestamp`,\n elb as `aws.elb.elb_name`,\n client_ip as `aws.elb.client.ip`,\n client_port as `aws.elb.client.port`,\n target_ip as `aws.elb.target_ip`,\n target_port as `aws.elb.target_port`,\n...

## Requirements * SessionStateMonitor execute in schedule interval. * Hang up job detection and recovery. * SessionStateMonitor check lastUpdateTime. * if it is not updated more than 15mins (configurable). Call...