dd-trace-py icon indicating copy to clipboard operation
dd-trace-py copied to clipboard

Published 20 hours ago • verified publisher icon DataDog

chore(llmobs): dac strip io from anthropic

Open jsimpher opened this issue 5 months ago • 3 comments

Remove potentially sensitive i/o data from apm spans. This way, prompt and completion data will only appear on the llm obs spans, which are/will be subject to data access controls.

Mostly, this just removes io tag sets. A few things (mostly metrics) have llmobs tags dependent on span tags, so there is a bit more refactoring there.

Let me know if I removed anything that should really stay, or if I missed something that should be restricted.

Checklist

  • [x] PR author has checked that all the criteria below are met
  • The PR description includes an overview of the change
  • The PR description articulates the motivation for the change
  • The change includes tests OR the PR description describes a testing strategy
  • The PR description notes risks associated with the change, if any
  • Newly-added code is easy to change
  • The change follows the library release note guidelines
  • The change includes or references documentation updates if necessary
  • Backport labels are set (if applicable)

Reviewer Checklist

  • [ ] Reviewer has checked that all the criteria below are met
  • Title is accurate
  • All changes are related to the pull request's stated goal
  • Avoids breaking API changes
  • Testing strategy adequately addresses listed risks
  • Newly-added code is easy to change
  • Release note makes sense to a user of the library
  • If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment
  • Backport labels are set in a manner that is consistent with the release branch maintenance policy

jsimpher avatar Jun 25 '25 14:06 jsimpher

CODEOWNERS have been resolved as:

releasenotes/notes/remove-io-data-from-span-anthropic-integration-a8b9ccdd3ac1d22d.yaml  @DataDog/apm-python
ddtrace/contrib/internal/anthropic/_streaming.py                        @DataDog/ml-observability
ddtrace/contrib/internal/anthropic/patch.py                             @DataDog/ml-observability
ddtrace/contrib/internal/anthropic/utils.py                             @DataDog/ml-observability
ddtrace/llmobs/_integrations/anthropic.py                               @DataDog/ml-observability
tests/contrib/anthropic/test_anthropic.py                               @DataDog/ml-observability
tests/snapshots/tests.contrib.anthropic.test_anthropic.test_anthropic_llm.json  @DataDog/ml-observability
tests/snapshots/tests.contrib.anthropic.test_anthropic.test_anthropic_llm_create_image.json  @DataDog/ml-observability
tests/snapshots/tests.contrib.anthropic.test_anthropic.test_anthropic_llm_error.json  @DataDog/ml-observability
tests/snapshots/tests.contrib.anthropic.test_anthropic.test_anthropic_llm_multiple_prompts.json  @DataDog/ml-observability
tests/snapshots/tests.contrib.anthropic.test_anthropic.test_anthropic_llm_multiple_prompts_with_chat_history.json  @DataDog/ml-observability
tests/snapshots/tests.contrib.anthropic.test_anthropic.test_anthropic_llm_stream.json  @DataDog/ml-observability
tests/snapshots/tests.contrib.anthropic.test_anthropic.test_anthropic_llm_stream_helper.json  @DataDog/ml-observability
tests/snapshots/tests.contrib.anthropic.test_anthropic.test_anthropic_llm_tools.json  @DataDog/ml-observability
tests/snapshots/tests.contrib.anthropic.test_anthropic.test_anthropic_llm_tools_full_use.json  @DataDog/ml-observability
tests/snapshots/tests.contrib.anthropic.test_anthropic.test_anthropic_llm_tools_full_use_stream.json  @DataDog/ml-observability
tests/snapshots/tests.contrib.anthropic.test_anthropic.test_anthropic_llm_tools_stream.json  @DataDog/ml-observability
tests/snapshots/tests.contrib.anthropic.test_anthropic.test_anthropic_llm_tools_stream_helper.json  @DataDog/ml-observability
tests/snapshots/tests.contrib.anthropic.test_anthropic.test_anthropic_llm_unserializable_arg.json  @DataDog/ml-observability

github-actions[bot] avatar Jun 25 '25 14:06 github-actions[bot]

Bootstrap import analysis

Comparison of import times between this PR and base.

Summary

The average import time from this PR is: 276 ± 4 ms.

The average import time from base is: 285 ± 6 ms.

The import time difference between this PR and base is: -8.7 ± 0.2 ms.

Import time breakdown

The following import paths have shrunk:

ddtrace.auto 3.727 ms (1.35%)
ddtrace.bootstrap.sitecustomize 2.530 ms (0.92%)
ddtrace.bootstrap.preload 2.053 ms (0.74%)
ddtrace.internal.remoteconfig.client 0.781 ms (0.28%)
multiprocessing.sharedctypes 0.160 ms (0.06%)
multiprocessing.heap 0.160 ms (0.06%)
mmap 0.160 ms (0.06%)
ddtrace.internal.remoteconfig.worker 0.033 ms (0.01%)
ddtrace.settings.code_origin 0.025 ms (0.01%)
ddtrace.appsec._common_module_patches 0.201 ms (0.07%)
ddtrace.appsec._asm_request_context 0.201 ms (0.07%)
ddtrace.appsec._utils 0.201 ms (0.07%)
ddtrace._trace.trace_handlers 0.099 ms (0.04%)
ddtrace._trace._inferred_proxy 0.035 ms (0.01%)
ddtrace.propagation.http 0.035 ms (0.01%)
ddtrace.internal._tagset 0.035 ms (0.01%)
ddtrace 1.197 ms (0.43%)
ddtrace._logger 0.180 ms (0.07%)
ddtrace.internal.telemetry 0.180 ms (0.07%)
ddtrace.internal.telemetry.writer 0.132 ms (0.05%)
http.client 0.132 ms (0.05%)
email.parser 0.132 ms (0.05%)
email.feedparser 0.132 ms (0.05%)
email._policybase 0.132 ms (0.05%)
email.header 0.132 ms (0.05%)
email.charset 0.132 ms (0.05%)
ddtrace.settings._agent 0.048 ms (0.02%)
socket 0.048 ms (0.02%)
_socket 0.048 ms (0.02%)
ddtrace.settings._config 0.171 ms (0.06%)
ddtrace.internal.gitmetadata 0.153 ms (0.06%)
ddtrace.ext.ci 0.153 ms (0.06%)
ddtrace.ext.git 0.153 ms (0.06%)
tempfile 0.153 ms (0.06%)
ddtrace.internal.schema 0.018 ms (0.01%)
ddtrace.internal._unpatched 0.133 ms (0.05%)
subprocess 0.099 ms (0.04%)
contextlib 0.099 ms (0.04%)
json 0.034 ms (0.01%)
json.decoder 0.034 ms (0.01%)
re 0.034 ms (0.01%)
enum 0.034 ms (0.01%)
types 0.034 ms (0.01%)
ddtrace.trace 0.058 ms (0.02%)

github-actions[bot] avatar Jun 25 '25 15:06 github-actions[bot]

Benchmarks

Benchmark execution time: 2025-07-03 16:24:28

Comparing candidate commit 739ee96d9e0570f2ebca385d7d92bf191e8c98e5 in PR branch jsimpher/dac-strip-io-from-anthropic with baseline commit 7b1d839f4f8278b7668118d403592484d1462aa8 in branch main.

Found 0 performance improvements and 1 performance regressions! Performance is the same for 499 metrics, 2 unstable metrics.

scenario:telemetryaddmetric-1-distribution-metric-1-times

  • 🟥 execution_time [+375.642ns; +401.040ns] or [+12.893%; +13.765%]

pr-commenter[bot] avatar Jun 25 '25 15:06 pr-commenter[bot]

/merge

jsimpher avatar Jul 03 '25 19:07 jsimpher

View all feedbacks in Devflow UI.

2025-07-03 19:10:00 UTC :information_source: Start processing command /merge

2025-07-03 19:10:09 UTC :information_source: MergeQueue: pull request added to the queue

The expected merge time in main is approximately 1h (p90).

2025-07-03 20:23:16 UTC :information_source: MergeQueue: This merge request was merged

dd-devflow[bot] avatar Jul 03 '25 19:07 dd-devflow[bot]