Support "data streams"

[phudson]

Elastic search has had data streams for a while now. However, the logging exporter doesn't send things in the right way.

According to here: https://www.elastic.co/guide/en/elasticsearch/reference/7.10/use-a-data-stream.html

"To add multiple documents with a single request, use the bulk API. Only create actions are supported."

curl -X PUT "localhost:9200/my-data-stream/_bulk?refresh&pretty" -H 'Content-Type: application/json' -d' {"create":{ }} { "@timestamp": "2099-03-08T11:04:05.000Z", "user": { "id": "vlb44hny" }, "message": "Login attempt failed" } {"create":{ }} { "@timestamp": "2099-03-08T11:06:07.000Z", "user": { "id": "8a4f500d" }, "message": "Login successful" } {"create":{ }} { "@timestamp": "2099-03-09T11:07:08.000Z", "user": { "id": "l7gk7f82" }, "message": "Logout successful" } ' But the exporter sends:

POST /wls/doc/_bulk?pretty HTTP/1.1 Accept: application/json Content-Type: application/json User-Agent: Jersey/2.22.4 (HttpUrlConnection 1.8.0_281) Host: localhost:9200 Connection: Keep-Alive Content-Length: 1512

{ "index" : { }} {"messageID": "BEA-2162611","message": "Creating ManagedScheduledExecutorService "DefaultManagedScheduledExecutorService" (partition="DOMAIN", module="null", application="bea_wls_deployment_internal", workmanager="default")","timestamp": 1641037065050,"serverName": "server_8080","threadName": "[ACTIVE] ExecuteThread: '10' for queue: 'weblogic.kernel.Default (self-tuning)'","severity": "Info","userId": "<WLS Kernel>","level": "Info","loggerName": "CONCURRENCY","formattedDate": "Jan 1, 2022 11:37:45,050 AM UTC","subSystem": "CONCURRENCY","machineName": "server","transactionId": "","diagnosticContextId": "7434236c-676a-4857-839f-9b4b00bc7fc7-000000a5","sequenceNumber": 5865,"domainUID": "domainid"}

and ES produces an error:

{ "took" : 0, "errors" : true, "items" : [ { "index" : { "_index" : "wls", "_type" : "doc", "_id" : null, "status" : 400, "error" : { "type" : "illegal_argument_exception", "reason" : "only write ops with an op_type of create are allowed in data streams" } } }, { "index" : { "_index" : "wls", "_type" : "doc", "_id" : null, "status" : 400, "error" : { "type" : "illegal_argument_exception", "reason" : "only write ops with an op_type of create are allowed in data streams" } } } ] }

Since (I assume) the exported never needs to update a doc, it could use "create" here. I'll try this out and submit a pull request if it works

[phudson]

This change was pretty easy:

In LogExportHandler.java, change

private static final String DOC_TYPE = "doc";
private static final String INDEX = " { \"index\" : { }} ";

to

private static final String DOC_TYPE = "_doc";
private static final String CREATE = " { \"create\" : { }} ";

and the use:

buffer.append(INDEX);

to

 buffer.append(CREATE);

And the name of the time stamp in the assembly of the event

    + dataAsJson("timestamp", wlLogRecord.getMillis())

to

    + dataAsJson("@timestamp", wlLogRecord.getMillis())

and in createMappings:

        + "        \"timestamp\": {"

to + " "@timestamp": {"

Note: The doc -> _doc change is a generally applicable bug fix, I think - the doc type should be _doc. In later ES versions, doc types is deprecated anyway, so it could be removed entirely. I didn't make that change.

The index->create change will also work for "straight" indicies as well as data streams, as far as I can see so this change too could be made unconditionally.

[phudson]

Nothing happened in a month. Is this project dead?