dd-trace-py icon indicating copy to clipboard operation
dd-trace-py copied to clipboard

chore(llmobs): dac strip io from vertex

Open jsimpher opened this issue 5 months ago • 4 comments

Remove potentially sensitive i/o data from apm spans. This way, prompt and completion data will only appear on the llm obs spans, which are/will be subject to data access controls.

Mostly, this just removes io tag sets. A few things (mostly metrics) have llmobs tags dependent on span tags, so there is a bit more refactoring there.

Let me know if I removed anything that should really stay, or if I missed something that should be restricted.

Checklist

  • [x] PR author has checked that all the criteria below are met
  • The PR description includes an overview of the change
  • The PR description articulates the motivation for the change
  • The change includes tests OR the PR description describes a testing strategy
  • The PR description notes risks associated with the change, if any
  • Newly-added code is easy to change
  • The change follows the library release note guidelines
  • The change includes or references documentation updates if necessary
  • Backport labels are set (if applicable)

Reviewer Checklist

  • [x] Reviewer has checked that all the criteria below are met
  • Title is accurate
  • All changes are related to the pull request's stated goal
  • Avoids breaking API changes
  • Testing strategy adequately addresses listed risks
  • Newly-added code is easy to change
  • Release note makes sense to a user of the library
  • If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment
  • Backport labels are set in a manner that is consistent with the release branch maintenance policy

jsimpher avatar Jun 17 '25 16:06 jsimpher

CODEOWNERS have been resolved as:

releasenotes/notes/remove-io-data-from-span-vertexai-integration-0ceacffc030e3f37.yaml  @DataDog/apm-python
ddtrace/contrib/internal/vertexai/_utils.py                             @DataDog/ml-observability
ddtrace/contrib/internal/vertexai/patch.py                              @DataDog/ml-observability
ddtrace/llmobs/_integrations/utils.py                                   @DataDog/ml-observability
ddtrace/llmobs/_integrations/vertexai.py                                @DataDog/ml-observability
tests/snapshots/tests.contrib.vertexai.test_vertexai.test_vertexai_completion.json  @DataDog/ml-observability
tests/snapshots/tests.contrib.vertexai.test_vertexai.test_vertexai_completion_error.json  @DataDog/ml-observability
tests/snapshots/tests.contrib.vertexai.test_vertexai.test_vertexai_completion_multiple_messages.json  @DataDog/ml-observability
tests/snapshots/tests.contrib.vertexai.test_vertexai.test_vertexai_completion_stream.json  @DataDog/ml-observability
tests/snapshots/tests.contrib.vertexai.test_vertexai.test_vertexai_completion_stream_error.json  @DataDog/ml-observability
tests/snapshots/tests.contrib.vertexai.test_vertexai.test_vertexai_completion_stream_tool.json  @DataDog/ml-observability
tests/snapshots/tests.contrib.vertexai.test_vertexai.test_vertexai_completion_system_prompt.json  @DataDog/ml-observability
tests/snapshots/tests.contrib.vertexai.test_vertexai.test_vertexai_completion_tool.json  @DataDog/ml-observability

github-actions[bot] avatar Jun 17 '25 16:06 github-actions[bot]

Bootstrap import analysis

Comparison of import times between this PR and base.

Summary

The average import time from this PR is: 278 ± 3 ms.

The average import time from base is: 281 ± 4 ms.

The import time difference between this PR and base is: -2.9 ± 0.2 ms.

Import time breakdown

The following import paths have shrunk:

ddtrace.auto 2.043 ms (0.74%)
ddtrace.bootstrap.sitecustomize 1.365 ms (0.49%)
ddtrace.bootstrap.preload 1.365 ms (0.49%)
ddtrace.internal.remoteconfig.client 0.669 ms (0.24%)
ddtrace 0.677 ms (0.24%)
ddtrace.internal._unpatched 0.031 ms (0.01%)
json 0.031 ms (0.01%)
json.decoder 0.031 ms (0.01%)
re 0.031 ms (0.01%)
enum 0.031 ms (0.01%)
types 0.031 ms (0.01%)

github-actions[bot] avatar Jun 17 '25 17:06 github-actions[bot]

Benchmarks

Benchmark execution time: 2025-07-01 19:11:33

Comparing candidate commit 70eebf837d4cc33d1abd8118f77d9e8a58c9f559 in PR branch jsimpher/dac-strip-io-from-vertex with baseline commit 329258474e6989e1299edfdb9ba13ff26b248fc5 in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 499 metrics, 3 unstable metrics.

pr-commenter[bot] avatar Jun 17 '25 17:06 pr-commenter[bot]

This should probably get a release note?

brettlangdon avatar Jun 25 '25 13:06 brettlangdon

/merge

jsimpher avatar Jul 01 '25 19:07 jsimpher

View all feedbacks in Devflow UI.

2025-07-01 19:50:47 UTC :information_source: Start processing command /merge


2025-07-01 19:50:57 UTC :information_source: MergeQueue: pull request added to the queue

The expected merge time in main is approximately 1h (p90).


2025-07-01 20:38:02 UTC :information_source: MergeQueue: This merge request was merged

dd-devflow[bot] avatar Jul 01 '25 19:07 dd-devflow[bot]