feat: As a user, I want to report APISIX's logs to Elasticsearch, so that i can check the logs with Kibana easily

[ccxhwmy]

Description

As a user, I want to report APISIX's logs to Elasticsearch, so that i can check the logs with Kibana easily.

Elasticsearch is the distributed search and analytics engine that provides a unified data store for solutions built on the Elastic Stack. According to the latest database search engine ranking [1], Elasticsearch has been ranked second Kibana provides access to the Elastic Observability, Security, and Enterprise Search solutions, enables you to interactively explore, visualize, and share insights into your data. APISIX report logs with Elasticsearch's Bulk API [2].

The following are the design and technical details:

1. Name

elasticsearch-logging

2. Configuration

{
    "endpoint": {
        "uri": "https://127.0.0.1:9200/<target>",
        "index": "service",
        "type": "collector",
        "username": "elastic",
        "password": "elastic",
        "ssl": true
    },
    "inactive_timeout": 10,
    "max_retry_count": 0,
    "buffer_duration": 60,
    "retry_delay": 1,
    "batch_max_size": 1
}

• endpoint Elasticsearch HTTP/HTTPS endpoint

• endpoint.uri Elasticsearch HTTP/HTTPS request uri, elasticsearch-logging will add "_bulk" at the end of it [3].

• endpoint.index Elasticsearch HTTP/HTTPS "_index" field [4]

• endpoint.type Elasticsearch HTTP/HTTPS "_type" field [5]

• endpoint.username Elasticsearch HTTP/HTTPS

• max_retry_count maximum number of retries before removing the entry from the processing pipeline when an error occurs.

• retry_delay number of seconds the process execution should be delayed if the execution fails.

• buffer_duration max age in seconds of the oldest entry in a batch before the batch must be processed

• inactive_timeout max age in seconds when the buffer will be flushed if inactive

• batch_max_size max size of each batch

3. Details

3.1 Configuration process

1. Get the username and password of Elasticsearch if needed.
2. Set the request URI, index, type(not necessary, Elasticsearch will config it with default config if it is null) to the plugin configuration.

3.2 HTTP Request process

1. Obtain and assemble request information data in the APISIX Log stage, data format refer to [6].
2. Add the assemble request data to the batch queue.
3. When the threshold of the batch queue is triggered, the requested data is subbmitted in batches to ElasticSearch.

[1] https://db-engines.com/en/ranking/search+engine [2] https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html#docs-bulk [3] https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html#docs-bulk-api-request [4] https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-index-field.html#mapping-index-field [5] https://www.elastic.co/guide/en/elasticsearch/reference/7.17/mapping-type-field.html#mapping-type-field [6] https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html#docs-bulk

[tzssangglass]

LGTM, would you plan to work on this?

[ccxhwmy]

LGTM, would you plan to work on this?

Yes, i'd like to.