feat(events): Add internal event id to event metadata

[ArunPiduguDD]

trafficstars

Overview

Adds an internal event id to all events that pass through Vector. This internal id will be a UUID generated on event creation, and can be used to uniquely identify an event as it traverses through different components in a pipeline.

Testing

Added unit test

Tested using demo_logs source type

[datadog-vectordotdev[bot]]

Datadog Report

Branch report: internal-event-id Commit report: acfc653 Test service: vector

:white_check_mark: 0 Failed, 7 Passed, 0 Skipped, 25.56s Total Time

[jorgehermo9]

Some test fail due to the fact that EventMetadata::Default() is now not deterministic (as uuid are random). Don't know what to do to fix that without involving much changes 🤔

Maybe a manual impl of PartialEq for that struct, so that the uuid field is ignored in the comparison...

[ArunPiduguDD]

Some test fail due to the fact that EventMetadata::Default() is now not deterministic (as uuid are random). Don't know what to do to fix that without involving much changes 🤔

Maybe a manual impl of PartialEq for that struct, so that the uuid field is ignored in the comparison...

@jorgehermo9 ended up using the derivative crate to ignore PartialEq (saw this crate is used in other parts of the vector codebase, but lmk if you think there might be any issues introduced by this)

[jorgehermo9]

yep, thats sounds like what I would do, lgtm

[jorgehermo9]

I think something went wrong with the merge and broke the diff... how did you do the merge with master?

[ArunPiduguDD]

I think something went wrong with the merge and broke the diff... how did you do the merge with master?

Strange, thought I just rebased on master, let me double check

[jorgehermo9]

I see that a lot of files were included in https://github.com/vectordotdev/vector/pull/21074/commits/a30e41792203f456f42142def56a1098fb0e9aad. Maybe should this PR be reviewed and approved again before including it in the merge queue? @bruceg

[ArunPiduguDD]

I see that a lot of files were included in a30e417. Maybe should this PR be reviewed and approved again before including it in the merge queue? @bruceg

Had a discussion with @bruceg and @jszwedko offline about regenerating these test fixtures, can wait for another stamp of approval before merging

[jszwedko]

Small nit: we generally like to discourage force-pushes to Vector PRs once reviews have started as it makes the PR history more difficult to understand. Instead of rebasing you can merge in master as needed. The PR is ultimately squashed into a single commit when merging so you don't have to worry about maintaining a clean history in the PR itself.

I'm realizing this isn't mentioned in https://github.com/vectordotdev/vector/blob/master/CONTRIBUTING.md. I'll add a note there.