beats Duplicated Google Workspace log entries by Filebeat

Many moths ago, we’ve noticed that some Google Workspace logs received by Filebeat got duplicated.

I’ve searched the internet for possible cause and find one similar issue here at Elastic Discuss, Google Workspace module using wrong field to avoid duplicates telling that "json.id.time", "json.id.uniqueQualifier", "json.id.applicationName", "json.id.customerId" are used to generate the _id.

After updating Filebeat to the most recent version (8.10.2 run from docker.elastic.co/beats/filebeat:8.10.2) I found that the same issue has different _id.

The issue was posted at forum https://discuss.elastic.co/t/duplicated-google-workspace-log-entries-by-filebeat/344374

Several days ago I've upgraded Filebeat to 8.14.0 and it didn't helped.

Here are two examples. First:

{
  "_index": "google_ws-2023.10.04",
  "_id": "4rrh-YoBiE7xzynem1Cm",
  "_source": { 
    "json": {
      "id": {
        "time": "2023-10-04T08:50:13.677Z"
      },
      "etag": "\"rQ3qpTrpjMqlOD9Fi6ZCgnpo6zAdUtM4Y4wU0J6c8Yw/UiNqGB-f4anaOLIVD9ya9Z-pAP0\"",
      "events": {},
      "actor": {}
    },
    "event": {
      "id": "-8909398197392254316",
      "created": "2023-10-04T08:50:25.347Z",
      "original": "{\"id\":{\"applicationName\":\"drive\",\"customerId\":\"C00hvn0vt\",\"time\":\"2023-10-04T08:50:13.677Z\",\"uniqueQualifier\":\"-8909398197392254316\"}"}"
    },
    "@timestamp": "2023-10-04T08:50:13.677Z",
  },
}

Second:

{
  "_index": "google_ws-2023.10.04",
  "_id": "QePm-YoBq7bjVLXLMFU_",
  "_source": {
    "json": {
      "id": {
        "time": "2023-10-04T08:50:13.677Z"
      },
      "etag": "\"rQ3qpTrpjMqlOD9Fi6ZCgnpo6zAdUtM4Y4wU0J6c8Yw/UiNqGB-f4anaOLIVD9ya9Z-pAP0\"",
      "events": {},
      "actor": {}
    },
    "event": {
      "created": "2023-10-04T08:55:25.376Z",
      "original": "{\"id\":{\"applicationName\":\"drive\",\"customerId\":\"C00hvn0vt\",\"time\":\"2023-10-04T08:50:13.677Z\",\"uniqueQualifier\":\"-8909398197392254316\"}"}"

    },
    "@timestamp": "2023-10-04T08:50:13.677Z",
  }
}

I.e. there is no uniqueQualifier, applicationName, customerId under the “json.id” key, as supposed to be, while they all still exists under the “event.original.id” key.

So could you please tell how this can be fixed?

Jun 11 '24 14:06 rlevytskyi

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

Jun 12 '24 18:06 elasticmachine

Hi @rlevytskyi, The deduplication fix for this was merged quite a while back in this PR based on this feedback. The unique id for deduplication is no longer in the json.id object but it's a finger print that lies in the _id field of the document. The main issue here seems to be it's generating unique _id's for duplicate events which is weird. Could you confirm if this is the case or not ?

Jun 22 '24 04:06 ShourieG

Yes I can confirm it's the case, different _id for the same event.

Jun 26 '24 11:06 rlevytskyi

After investigation of this and similar issues, we've observed the following:

Duplicate issues were significantly reduced and fixed for the most part after this PR was merged.
Duplication issues seem to be more relevant with the workspace module when compared with the workspace integration.
The google workspace module uses a fingerprint processor that does not support canonical ordering of the event object keys, this was recently fixed with this PR and should help reduce duplication going forward.
The duplication issue talked about in this current issue seems to stem from issues outside our control and the involvement of Logstash or some issue that is causing the ingest pipeline to not work as expected as the presence of the "_source" object in the resulting documents suggest that the pipeline did not remove them correctly. Also fields inside the "_source" are missing which is leading to different fingerprints for the same document.

We are keeping this issue open to see if the duplication issue persists following the recent PR fix and we will also introduce an enhancement for adding conditional canonical sorting of keys to the fingerprint processor soon. cc: @narph

Jul 04 '24 09:07 ShourieG

Thank you! I'll test once it will be merged to the new version.

Jul 04 '24 11:07 rlevytskyi

Just updated filebeat to 8.14.3 and don't see duplicated messages now. Will update you guys on Monday.

Jul 26 '24 10:07 rlevytskyi

@rlevytskyi , any updates here? can we close the issue?

Aug 14 '24 07:08 narph

Sorry for the delay, I see no duplicates now. Filebeat works properly.

Aug 15 '24 05:08 rlevytskyi

It happened again.

Sep 02 '24 13:09 rlevytskyi

The issue is still exist

Sep 02 '24 13:09 maksimsaroka

{
  "_index": "google_ws-2024.35",
  "_id": "WQj5m5EBYPKiM--Ll6eM",
  "_version": 1,
  "_score": null,
  "_source": {
    "office_name": "nl2",
    "google_workspace": {
      "kind": "admin#reports#activity",
      "login": {
        "challenge_method": [
          "none"
        ],
        "is_suspicious": false,
        "type": "reauth"
      },
      "event": {
        "type": "login"
      }
    },
    "source": {
      "ip": "76.185.108.167",
      "user": {
        "id": "113905123137720729621",
        "name": "mboustridge",
        "email": "[email protected]",
        "domain": "exadel.com"
      }
    },
    "user": {
      "name": "mboustridge",
      "id": "113905123137720729621",
      "domain": "exadel.com"
    },
    "ecs": {},
    "service": {
      "type": "google_workspace"
    },
    "fileset": {
      "name": "login"
    },
    "json": {
      "actor": {},
      "id": {
        "time": "2024-08-29T01:00:48.152Z"
      },
      "etag": "\"JX6NVNYagoMncd_ui5IaKzerPU54GzNJg5WsuXIkYzw/H6NnWqcZznaFd6xmuxV63yJUDpM\"",
      "events": {}
    },
    "agent": {
      "name": "k8s-ams-filebeat"
    },
    "event": {
      "id": "-9198098292682564254",
      "dataset": "google_workspace.login",
      "provider": "login",
      "original": "{\"actor\":{\"email\":\"[email protected]\",\"profileId\":\"113905123137720729621\"},\"etag\":\"\\\"JX6NVNYagoMncd_ui5IaKzerPU54GzNJg5WsuXIkYzw/H6NnWqcZznaFd6xmuxV63yJUDpM\\\"\",\"events\":{\"name\":\"login_success\",\"parameters\":[{\"name\":\"login_type\",\"value\":\"reauth\"},{\"multiValue\":[\"none\"],\"name\":\"login_challenge_method\"},{\"boolValue\":false,\"name\":\"is_suspicious\"}],\"type\":\"login\"},\"id\":{\"applicationName\":\"login\",\"customerId\":\"C00hvn0vt\",\"time\":\"2024-08-29T01:00:48.152Z\",\"uniqueQualifier\":\"-9198098292682564254\"},\"ipAddress\":\"76.185.108.167\",\"kind\":\"admin#reports#activity\"}",
      "category": [
        "authentication",
        "session"
      ],
      "action": "login_success",
      "created": "2024-08-29T02:31:46.568Z",
      "module": "google_workspace",
      "type": [
        "start"
      ],
      "outcome": "success"
    },
    "type": "logs",
    "@timestamp": "2024-08-29T01:00:48.152Z",
    "related": {
      "ip": [
        "76.185.108.167"
      ],
      "user": [
        "mboustridge"
      ]
    },
    "@version": "1",
    "organization": {
      "id": "C00hvn0vt"
    },
    "input": {},
    "tags": [
      "forwarded",
      "logstash-cv107",
      "logstash-k8s"
    ]
  },
  "fields": {
    "json.id.time": [
      "2024-08-29T01:00:48.152Z"
    ],
    "@timestamp": [
      "2024-08-29T01:00:48.152Z"
    ],
    "event.created": [
      "2024-08-29T02:31:46.568Z"
    ]
  },
  "highlight": {
    "event.original": [
      "{\"@opensearch-dashboards-highlighted-field@actor@/opensearch-dashboards-highlighted-field@\":{\"@opensearch-dashboards-highlighted-field@email@/opensearch-dashboards-highlighted-field@\":\"@opensearch-dashboards-highlighted-field@mboustridge@/opensearch-dashboards-highlighted-field@@@[email protected]@/opensearch-dashboards-highlighted-field@\",\"@opensearch-dashboards-highlighted-field@profileId@/opensearch-dashboards-highlighted-field@\":\"@opensearch-dashboards-highlighted-field@113905123137720729621@/opensearch-dashboards-highlighted-field@\"},\"@opensearch-dashboards-highlighted-field@etag@/opensearch-dashboards-highlighted-field@\":\"\\\"@opensearch-dashboards-highlighted-field@JX6NVNYagoMncd_ui5IaKzerPU54GzNJg5WsuXIkYzw@/opensearch-dashboards-highlighted-field@/@opensearch-dashboards-highlighted-field@H6NnWqcZznaFd6xmuxV63yJUDpM@/opensearch-dashboards-highlighted-field@\\\"\",\"@opensearch-dashboards-highlighted-field@events@/opensearch-dashboards-highlighted-field@\":{\"@opensearch-dashboards-highlighted-field@name@/opensearch-dashboards-highlighted-field@\":\"@opensearch-dashboards-highlighted-field@login_success@/opensearch-dashboards-highlighted-field@\",\"@opensearch-dashboards-highlighted-field@parameters@/opensearch-dashboards-highlighted-field@\":[{\"@opensearch-dashboards-highlighted-field@name@/opensearch-dashboards-highlighted-field@\":\"@opensearch-dashboards-highlighted-field@login_type@/opensearch-dashboards-highlighted-field@\",\"@opensearch-dashboards-highlighted-field@value@/opensearch-dashboards-highlighted-field@\":\"@opensearch-dashboards-highlighted-field@reauth@/opensearch-dashboards-highlighted-field@\"},{\"@opensearch-dashboards-highlighted-field@multiValue@/opensearch-dashboards-highlighted-field@\":[\"@opensearch-dashboards-highlighted-field@none@/opensearch-dashboards-highlighted-field@\"],\"@opensearch-dashboards-highlighted-field@name@/opensearch-dashboards-highlighted-field@\":\"@opensearch-dashboards-highlighted-field@login_challenge_method@/opensearch-dashboards-highlighted-field@\"},{\"@opensearch-dashboards-highlighted-field@boolValue@/opensearch-dashboards-highlighted-field@\":@opensearch-dashboards-highlighted-field@false@/opensearch-dashboards-highlighted-field@,\"@opensearch-dashboards-highlighted-field@name@/opensearch-dashboards-highlighted-field@\":\"@opensearch-dashboards-highlighted-field@is_suspicious@/opensearch-dashboards-highlighted-field@\"}],\"@opensearch-dashboards-highlighted-field@type@/opensearch-dashboards-highlighted-field@\":\"@opensearch-dashboards-highlighted-field@login@/opensearch-dashboards-highlighted-field@\"},\"@opensearch-dashboards-highlighted-field@id@/opensearch-dashboards-highlighted-field@\":{\"@opensearch-dashboards-highlighted-field@applicationName@/opensearch-dashboards-highlighted-field@\":\"@opensearch-dashboards-highlighted-field@login@/opensearch-dashboards-highlighted-field@\",\"@opensearch-dashboards-highlighted-field@customerId@/opensearch-dashboards-highlighted-field@\":\"@opensearch-dashboards-highlighted-field@C00hvn0vt@/opensearch-dashboards-highlighted-field@\",\"@opensearch-dashboards-highlighted-field@time@/opensearch-dashboards-highlighted-field@\":\"@opensearch-dashboards-highlighted-field@2024@/opensearch-dashboards-highlighted-field@-@opensearch-dashboards-highlighted-field@08@/opensearch-dashboards-highlighted-field@-@opensearch-dashboards-highlighted-field@29T01@/opensearch-dashboards-highlighted-field@:@opensearch-dashboards-highlighted-field@00@/opensearch-dashboards-highlighted-field@:@[email protected]@/opensearch-dashboards-highlighted-field@\",\"@opensearch-dashboards-highlighted-field@uniqueQualifier@/opensearch-dashboards-highlighted-field@\":\"-@opensearch-dashboards-highlighted-field@9198098292682564254@/opensearch-dashboards-highlighted-field@\"},\"@opensearch-dashboards-highlighted-field@ipAddress@/opensearch-dashboards-highlighted-field@\":\"@[email protected]@/opensearch-dashboards-highlighted-field@\",\"@opensearch-dashboards-highlighted-field@kind@/opensearch-dashboards-highlighted-field@\":\"@opensearch-dashboards-highlighted-field@admin@/opensearch-dashboards-highlighted-field@#@opensearch-dashboards-highlighted-field@reports@/opensearch-dashboards-highlighted-field@#@opensearch-dashboards-highlighted-field@activity@/opensearch-dashboards-highlighted-field@\"}"
    ]
  },
  "sort": [
    1724893248152
  ]
}

and this

{
  "_index": "google_ws-2024.35",
  "_id": "Ag7-m5EBYPKiM--LLzV_",
  "_version": 1,
  "_score": null,
  "_source": {
    "agent": {
      "name": "k8s-ams-filebeat"
    },
    "input": {},
    "related": {
      "ip": [
        "76.185.108.167"
      ],
      "user": [
        "mboustridge"
      ]
    },
    "tags": [
      "forwarded",
      "logstash-cv107",
      "logstash-k8s"
    ],
    "user": {
      "id": "113905123137720729621",
      "name": "mboustridge",
      "domain": "exadel.com"
    },
    "source": {
      "ip": "76.185.108.167",
      "user": {
        "id": "113905123137720729621",
        "name": "mboustridge",
        "email": "[email protected]",
        "domain": "exadel.com"
      }
    },
    "google_workspace": {
      "login": {
        "is_suspicious": false,
        "challenge_method": [
          "none"
        ],
        "type": "reauth"
      },
      "event": {
        "type": "login"
      },
      "kind": "admin#reports#activity"
    },
    "@timestamp": "2024-08-29T01:00:48.152Z",
    "json": {
      "id": {
        "time": "2024-08-29T01:00:48.152Z"
      },
      "etag": "\"JX6NVNYagoMncd_ui5IaKzerPU54GzNJg5WsuXIkYzw/H6NnWqcZznaFd6xmuxV63yJUDpM\"",
      "events": {},
      "actor": {}
    },
    "event": {
      "id": "-9198098292682564254",
      "dataset": "google_workspace.login",
      "provider": "login",
      "category": [
        "authentication",
        "session"
      ],
      "action": "login_success",
      "module": "google_workspace",
      "type": [
        "start"
      ],
      "original": "{\"actor\":{\"email\":\"[email protected]\",\"profileId\":\"113905123137720729621\"},\"etag\":\"\\\"JX6NVNYagoMncd_ui5IaKzerPU54GzNJg5WsuXIkYzw/H6NnWqcZznaFd6xmuxV63yJUDpM\\\"\",\"events\":{\"name\":\"login_success\",\"parameters\":[{\"name\":\"login_type\",\"value\":\"reauth\"},{\"multiValue\":[\"none\"],\"name\":\"login_challenge_method\"},{\"boolValue\":false,\"name\":\"is_suspicious\"}],\"type\":\"login\"},\"id\":{\"applicationName\":\"login\",\"customerId\":\"C00hvn0vt\",\"time\":\"2024-08-29T01:00:48.152Z\",\"uniqueQualifier\":\"-9198098292682564254\"},\"ipAddress\":\"76.185.108.167\",\"kind\":\"admin#reports#activity\"}",
      "created": "2024-08-29T02:36:46.556Z",
      "outcome": "success"
    },
    "office_name": "nl2",
    "service": {
      "type": "google_workspace"
    },
    "fileset": {
      "name": "login"
    },
    "@version": "1",
    "organization": {
      "id": "C00hvn0vt"
    },
    "type": "logs",
    "ecs": {}
  },
  "fields": {
    "json.id.time": [
      "2024-08-29T01:00:48.152Z"
    ],
    "@timestamp": [
      "2024-08-29T01:00:48.152Z"
    ],
    "event.created": [
      "2024-08-29T02:36:46.556Z"
    ]
  },
  "highlight": {
    "event.original": [
      "{\"@opensearch-dashboards-highlighted-field@actor@/opensearch-dashboards-highlighted-field@\":{\"@opensearch-dashboards-highlighted-field@email@/opensearch-dashboards-highlighted-field@\":\"@opensearch-dashboards-highlighted-field@mboustridge@/opensearch-dashboards-highlighted-field@@@[email protected]@/opensearch-dashboards-highlighted-field@\",\"@opensearch-dashboards-highlighted-field@profileId@/opensearch-dashboards-highlighted-field@\":\"@opensearch-dashboards-highlighted-field@113905123137720729621@/opensearch-dashboards-highlighted-field@\"},\"@opensearch-dashboards-highlighted-field@etag@/opensearch-dashboards-highlighted-field@\":\"\\\"@opensearch-dashboards-highlighted-field@JX6NVNYagoMncd_ui5IaKzerPU54GzNJg5WsuXIkYzw@/opensearch-dashboards-highlighted-field@/@opensearch-dashboards-highlighted-field@H6NnWqcZznaFd6xmuxV63yJUDpM@/opensearch-dashboards-highlighted-field@\\\"\",\"@opensearch-dashboards-highlighted-field@events@/opensearch-dashboards-highlighted-field@\":{\"@opensearch-dashboards-highlighted-field@name@/opensearch-dashboards-highlighted-field@\":\"@opensearch-dashboards-highlighted-field@login_success@/opensearch-dashboards-highlighted-field@\",\"@opensearch-dashboards-highlighted-field@parameters@/opensearch-dashboards-highlighted-field@\":[{\"@opensearch-dashboards-highlighted-field@name@/opensearch-dashboards-highlighted-field@\":\"@opensearch-dashboards-highlighted-field@login_type@/opensearch-dashboards-highlighted-field@\",\"@opensearch-dashboards-highlighted-field@value@/opensearch-dashboards-highlighted-field@\":\"@opensearch-dashboards-highlighted-field@reauth@/opensearch-dashboards-highlighted-field@\"},{\"@opensearch-dashboards-highlighted-field@multiValue@/opensearch-dashboards-highlighted-field@\":[\"@opensearch-dashboards-highlighted-field@none@/opensearch-dashboards-highlighted-field@\"],\"@opensearch-dashboards-highlighted-field@name@/opensearch-dashboards-highlighted-field@\":\"@opensearch-dashboards-highlighted-field@login_challenge_method@/opensearch-dashboards-highlighted-field@\"},{\"@opensearch-dashboards-highlighted-field@boolValue@/opensearch-dashboards-highlighted-field@\":@opensearch-dashboards-highlighted-field@false@/opensearch-dashboards-highlighted-field@,\"@opensearch-dashboards-highlighted-field@name@/opensearch-dashboards-highlighted-field@\":\"@opensearch-dashboards-highlighted-field@is_suspicious@/opensearch-dashboards-highlighted-field@\"}],\"@opensearch-dashboards-highlighted-field@type@/opensearch-dashboards-highlighted-field@\":\"@opensearch-dashboards-highlighted-field@login@/opensearch-dashboards-highlighted-field@\"},\"@opensearch-dashboards-highlighted-field@id@/opensearch-dashboards-highlighted-field@\":{\"@opensearch-dashboards-highlighted-field@applicationName@/opensearch-dashboards-highlighted-field@\":\"@opensearch-dashboards-highlighted-field@login@/opensearch-dashboards-highlighted-field@\",\"@opensearch-dashboards-highlighted-field@customerId@/opensearch-dashboards-highlighted-field@\":\"@opensearch-dashboards-highlighted-field@C00hvn0vt@/opensearch-dashboards-highlighted-field@\",\"@opensearch-dashboards-highlighted-field@time@/opensearch-dashboards-highlighted-field@\":\"@opensearch-dashboards-highlighted-field@2024@/opensearch-dashboards-highlighted-field@-@opensearch-dashboards-highlighted-field@08@/opensearch-dashboards-highlighted-field@-@opensearch-dashboards-highlighted-field@29T01@/opensearch-dashboards-highlighted-field@:@opensearch-dashboards-highlighted-field@00@/opensearch-dashboards-highlighted-field@:@[email protected]@/opensearch-dashboards-highlighted-field@\",\"@opensearch-dashboards-highlighted-field@uniqueQualifier@/opensearch-dashboards-highlighted-field@\":\"-@opensearch-dashboards-highlighted-field@9198098292682564254@/opensearch-dashboards-highlighted-field@\"},\"@opensearch-dashboards-highlighted-field@ipAddress@/opensearch-dashboards-highlighted-field@\":\"@[email protected]@/opensearch-dashboards-highlighted-field@\",\"@opensearch-dashboards-highlighted-field@kind@/opensearch-dashboards-highlighted-field@\":\"@opensearch-dashboards-highlighted-field@admin@/opensearch-dashboards-highlighted-field@#@opensearch-dashboards-highlighted-field@reports@/opensearch-dashboards-highlighted-field@#@opensearch-dashboards-highlighted-field@activity@/opensearch-dashboards-highlighted-field@\"}"
    ]
  },
  "sort": [
    1724893248152
  ]
}

Sep 02 '24 13:09 rlevytskyi

Hello Guys,

Any progress on this?

Nov 07 '24 11:11 msaroka-hpe

Hello Guys, Do you have any updates for us?

Mar 13 '25 11:03 maksimsaroka

We've turned out that Filebeat doesn’t pass metadata while sending to Logstash.

I have to pass ID as a separate data field and then recreate metadata while ingesting to Opensearch.

No more duplicates.

@narph the issue can be closed

Mar 21 '25 07:03 maksimsaroka

@maksimsaroka thank you for looking into this

Mar 25 '25 08:03 narph