feat(vrl): add functions for internal vector metrics access in VRL

[esensar]

Summary

Adds 3 new functions: get_vector_metric, find_vector_metrics and aggregate_vector_metrics, enabling access to current Vector metrics (with some delay, to reduce locking).

Vector configuration

api:
  enabled: true
  address: "0.0.0.0:8686"

metrics_storage_refresh_period: 1

sources:
  demo_logs:
    type: demo_logs
    format: "json"

transforms:
  remap_vrl_metrics:
    type: remap
    inputs: ["demo_logs"]
    source: |
      .utilization = find_vector_metrics("utilization", tags: { "component_id": "console" })
      .utilization_sum = aggregate_vector_metrics("sum", "utilization")
      .utilization_avg = aggregate_vector_metrics("avg", "utilization")
      .utilization_min = aggregate_vector_metrics("min", "utilization")
      .utilization_max = aggregate_vector_metrics("max", "utilization")

sinks:
  console:
    inputs: ["remap_vrl_metrics"]
    type: console
    encoding:
      codec: json

How did you test this PR?

Ran Vector with the above configuration.

Change Type

• [ ] Bug fix
• [x] New feature
• [ ] Non-functional (chore, refactoring, docs)
• [ ] Performance

Is this a breaking change?

• [ ] Yes
• [x] No

Does this PR include user facing changes?

• [x] Yes. Please add a changelog fragment based on our guidelines.
• [ ] No. A maintainer will apply the no-changelog label to this PR.

References

• Closes: #23284

Notes

• Please read our Vector contributor resources.
• Do not hesitate to use @vectordotdev/vector to reach out to us regarding this PR.
• Some CI checks run only after we manually approve them.
  • We recommend adding a pre-push hook, please see this template.
  • Alternatively, we recommend running the following locally before pushing to the remote branch:
    • cargo fmt --all
    • cargo clippy --workspace --all-targets -- -D warnings
    • cargo nextest run --workspace (alternatively, you can run cargo test --all)
• After a review is requested, please avoid force pushes to help us review incrementally.
  • Feel free to push as many commits as you want. They will be squashed into one before merging.
  • For example, you can run git merge origin master and git push.
• If this PR introduces changes Vector dependencies (modifies Cargo.lock), please run cargo vdev build licenses to regenerate the license inventory and commit the changes (if any). More details here.

---

Sponsored by Quad9

[esensar]

@pront this is still not done (I need to implement filtering by tags and maybe aggregation too), but I wanted to check if the way this global metrics cache is plugged into the topology looks alright to you.

[esensar]

@pront Just checking if you had time to look at this one.

[pront]

@pront Just checking if you had time to look at this one.

Hi @esensar, no we tend to not review draft PRs. In the future if you want us to take look please mark it as "ready for review" and ping @vectordotdev/vector

[esensar]

@pront Just checking if you had time to look at this one.

Hi @esensar, no we tend to not review draft PRs. In the future if you want us to take look please mark it as "ready for review" and ping @vectordotdev/vector

Oh, sorry about that. I left it as a draft because it is not really complete. It is missing the additional function arguments, for filtering by tags and similar, but I was wondering if the way I hooked up the metrics storage with the topology was alright. I can add these missing arguments and finalize the functions if you would prefer that to be done first.

[pront]

@pront Just checking if you had time to look at this one.

Hi @esensar, no we tend to not review draft PRs. In the future if you want us to take look please mark it as "ready for review" and ping @vectordotdev/vector

Oh, sorry about that. I left it as a draft because it is not really complete. It is missing the additional function arguments, for filtering by tags and similar, but I was wondering if the way I hooked up the metrics storage with the topology was alright. I can add these missing arguments and finalize the functions if you would prefer that to be done first.

No problem. Just to manage expectations, I don't think we have enough time to put in this release. But will prioritize it for the next one.

[esensar]

Oh, sorry about that. I left it as a draft because it is not really complete. It is missing the additional function arguments, for filtering by tags and similar, but I was wondering if the way I hooked up the metrics storage with the topology was alright. I can add these missing arguments and finalize the functions if you would prefer that to be done first.

No problem. Just to manage expectations, I don't think we have enough time to put in this release. But will prioritize it for the next one.

Yeah, understood, that is fine. I am pretty sure I won't have it ready for the release anyways.

[esensar]

@pront this is finally ready for review now.

[esensar]

Just a reminder that this is ready for review. It also includes a minor change in how transforms are built, meaning this PR can easily get conflicts (it already did a couple of times in the process).

[esensar]

@pront Sorry for pinging you directly, I can't seem to ping @vectordotdev/vector This one has been ready for review for a while now, so just checking if there is time on your side to take a look at it.

[pront]

@pront Sorry for pinging you directly, I can't seem to ping @vectordotdev/vector This one has been ready for review for a while now, so just checking if there is time on your side to take a look at it.

Apologies for the delay, I am (and will be) on a leave for a while. The Vector ping seemed to work for me though. Thanks for flagging though, we want folks to easily ping here.

[esensar]

I have fixed the one about using all() from vrl stdlib. Other tests passed for me.

[thomasqueirozb]

Other tests passed for me.

cd lib/vector-vrl-metrics && cargo nextest run is failing for me. Letting CI run to see if it's replicable.

[thomasqueirozb]

Looks like tests are failing in the CI also, not sure if something is different with your local environment that is making these tests pass.

[thomasqueirozb]

It actually looks like playground is failing. I think that maybe we need to omit this new crate from vector_vrl_functions::all on wasm

[esensar]

It actually looks like playground is failing. I think that maybe we need to omit this new crate from vector_vrl_functions::all on wasm

Yeah, I did that. It should be working now (it worked locally :smile: -- wasm-pack build --target web --out-dir public/pkg in lib/vector-vrl/web-playground).