Map Jenkins Authentication Otel Log attributes to Elastic ECS

[cyrille-leclerc]

What feature do you want to see added?

Map Jenkins Authentication Otel Log attributes to Elastic ECS when ingesting Jenkins Otel logs in Elastic to better integrate with Elastic SIEM.

Jenkins OTel Log Attribute	Standard OTel Semantic Convention	Elastic ECS Field
enduser.id=hudson.model.User.getId()	yes	TBD user.id ("Unique identifier of the user") or user.name ("Short name or login of the user")
net.peer.ip=request.getRemoteAddr()	yes
event.action='user_login'	no	event.action
event.category='authentication'	no	event.category
event.outcome='success' or 'failure'	no	event.outcome

❓ Pending questions:

• Should Elastic map Jenkins Otel Logs attributes if they are not part of the official Otel Semantic Conventions? Should Elastic map any Otel attribute that match an equivalent official ECS field? For example should Elastic automatically map the an OpenTelemetry attribute event.action to Elastic event.action or should it map it to labels.event.action? Currently it's the latter.

Elastic SIEM detections:

• Elastic Security / Detect Unusual Login Activity
• Auditbeat System Login Dataset

Example Successful Authentication Log Message

{
  "_index": ".ds-logs-apm.app-default-2022.03.07-000002",
  "_id": "ajqBq38B8sassxURpvo6",
  "_version": 1,
  "_score": 1,
  "_source": {
    "container": {
      "id": "559d4b8ac80f638fbc37f1804394a24d1a5f3fadd567054995b586a226055253"
    },
    "agent": {
      "name": "opentelemetry/java",
      "version": "1.12.0"
    },
    "data_stream.namespace": "default",
    "message": "Successful login of user 'admin' from 176.175.74.234",
    "processor": {
      "name": "log",
      "event": "log"
    },
    "data_stream.type": "logs",
    "labels": {
      "process_runtime_description": "Eclipse Adoptium OpenJDK 64-Bit Server VM 11.0.14+9",
      "event_action": "user_login",
      "jenkins_url": "https://jenkins.104.197.117.206.ip.es.io/",
      "enduser_id": "admin",
      "event_outcome": "success",
      "jenkins_version": "2.319.3",
      "net_peer_ip": "176.175.74.234",
      "service_namespace": "jenkins",
      "jenkins_opentelemetry_plugin_version": "2.3.0-rc1",
      "event_category": "authentication"
    },
    "observer": {
      "hostname": "8bacbea9c748",
      "id": "8ed90c13-d060-42db-a432-ad699ad6bb20",
      "ephemeral_id": "cfcf472e-eaf4-4670-9406-e904441836c1",
      "type": "apm-server",
      "version": "8.1.0",
      "version_major": 8
    },
    "@timestamp": "2022-03-21T08:04:33.856Z",
    "ecs": {
      "version": "1.12.0"
    },
    "service": {
      "node": {
        "name": "559d4b8ac80f638fbc37f1804394a24d1a5f3fadd567054995b586a226055253"
      },
      "name": "jenkins",
      "runtime": {
        "name": "OpenJDK Runtime Environment",
        "version": "11.0.14+9"
      },
      "language": {
        "name": "java"
      },
      "version": "2.319.3"
    },
    "data_stream.dataset": "apm.app",
    "host": {
      "hostname": "jenkins-0",
      "os": {
        "type": "linux",
        "platform": "linux",
        "full": "Linux 5.4.170+"
      },
      "name": "jenkins-0",
      "architecture": "amd64"
    },
    "event": {
      "severity": 9,
      "agent_id_status": "missing",
      "ingested": "2022-03-21T08:04:35Z"
    }
  },
  "fields": {
    "labels.jenkins_version": [
      "2.319.3"
    ],
    "labels.net_peer_ip": [
      "176.175.74.234"
    ],
    "labels.process_runtime_description": [
      "Eclipse Adoptium OpenJDK 64-Bit Server VM 11.0.14+9"
    ],
    "host.os.full": [
      "Linux 5.4.170+"
    ],
    "labels.enduser_id": [
      "admin"
    ],
    "labels.event_action": [
      "user_login"
    ],
    "labels.jenkins_url": [
      "https://jenkins.104.197.117.206.ip.es.io/"
    ],
    "service.node.name": [
      "559d4b8ac80f638fbc37f1804394a24d1a5f3fadd567054995b586a226055253"
    ],
    "host.hostname": [
      "jenkins-0"
    ],
    "service.language.name": [
      "java"
    ],
    "container.id": [
      "559d4b8ac80f638fbc37f1804394a24d1a5f3fadd567054995b586a226055253"
    ],
    "processor.event": [
      "log"
    ],
    "agent.name": [
      "opentelemetry/java"
    ],
    "host.name": [
      "jenkins-0"
    ],
    "event.agent_id_status": [
      "missing"
    ],
    "labels.event_outcome": [
      "success"
    ],
    "event.severity": [
      9
    ],
    "service.name": [
      "jenkins"
    ],
    "host.os.type": [
      "linux"
    ],
    "data_stream.namespace": [
      "default"
    ],
    "processor.name": [
      "log"
    ],
    "service.runtime.name": [
      "OpenJDK Runtime Environment"
    ],
    "service.runtime.version": [
      "11.0.14+9"
    ],
    "observer.version_major": [
      8
    ],
    "message": [
      "Successful login of user 'admin' from 176.175.74.234"
    ],
    "observer.hostname": [
      "8bacbea9c748"
    ],
    "data_stream.type": [
      "logs"
    ],
    "labels.jenkins_opentelemetry_plugin_version": [
      "2.3.0-rc1"
    ],
    "host.architecture": [
      "amd64"
    ],
    "event.ingested": [
      "2022-03-21T08:04:35.000Z"
    ],
    "observer.id": [
      "8ed90c13-d060-42db-a432-ad699ad6bb20"
    ],
    "@timestamp": [
      "2022-03-21T08:04:33.856Z"
    ],
    "service.version": [
      "2.319.3"
    ],
    "observer.ephemeral_id": [
      "cfcf472e-eaf4-4670-9406-e904441836c1"
    ],
    "observer.version": [
      "8.1.0"
    ],
    "host.os.platform": [
      "linux"
    ],
    "observer.type": [
      "apm-server"
    ],
    "ecs.version": [
      "1.12.0"
    ],
    "data_stream.dataset": [
      "apm.app"
    ],
    "agent.version": [
      "1.12.0"
    ],
    "labels.event_category": [
      "authentication"
    ],
    "labels.service_namespace": [
      "jenkins"
    ]
  }
}

Upstream changes

No response

[MikePaquette]

Hi @cyrille-leclerc, please let me offer a couple of suggestions:

Should Elastic map Jenkins Otel Logs attributes if they are not part of the official Otel Semantic Conventions? Should Elastic map any Otel attribute that match an equivalent official ECS field?

If, as the initial comment suggests, the goal is to "Map Jenkins Authentication Otel Log attributes to Elastic ECS when ingesting Jenkins Otel logs in Elastic to better integrate with Elastic SIEM.", then yes, an optimal approach would be to attempt to map all fields from the Jenkins Otel Logs to ECS, not limiting mapping to just those fields for which Standard OTel Semantic Convention exists.

For example should Elastic automatically map the an OpenTelemetry attribute event.action to Elastic event.action or should it map it to labels.event.action? Currently it's the latter.

When using ECS, optimal analyst experiences (single click filtering, aggregations, single-click-pivoting) become available when the ECS fields are present in their specified hierarchy within the ECS namespace. Which means that event.* fields such as event.category, event.outcome, and event.action should be present at the root of the event document's namespace, and NOT nested under labels.*