dd-trace-py inferred service for auto injection

Description

This PR adds another layer to service naming. Instead of defaulting to unnamed-python-service when no DD_SERVICE or programatic service name was provided, this change attempts to find a package or module name.

The algo:

First it will try to find a package name via a metadata file (setup.py, pyproject.toml). It will begin looking for these metadata files from the entrypoint directory used to start the program.
If no package name was found, we will use the entrypoint script / module name used to start the process. If we were not able to resolve this entrypoint, global service will default to unnamed-python-service.

Checklist

[ ] Change(s) are motivated and described in the PR description
[ ] Testing strategy is described if automated tests are not included in the PR
[ ] Risks are described (performance impact, potential for breakage, maintainability)
[ ] Change is maintainable (easy to change, telemetry, documentation)
[ ] Library release note guidelines are followed or label changelog/no-changelog is set
[ ] Documentation is included (in-code, generated user docs, public corp docs)
[ ] Backport labels are set (if applicable)
[ ] If this PR changes the public interface, I've notified @DataDog/apm-tees.

Reviewer Checklist

[ ] Title is accurate
[ ] All changes are related to the pull request's stated goal
[ ] Description motivates each change
[ ] Avoids breaking API changes
[ ] Testing strategy adequately addresses listed risks
[ ] Change is maintainable (easy to change, telemetry, documentation)
[ ] Release note makes sense to a user of the library
[ ] Author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment
[ ] Backport labels are set in a manner that is consistent with the release branch maintenance policy

May 24 '24 13:05 wconti27

Datadog Report

Branch report: conti/inferred-service-autoinjection Commit report: cbc0a71 Test service: dd-trace-py

:x: 874 Failed (6 Known Flaky), 130072 Passed, 42839 Skipped, 6h 9m 36.12s Total duration (3h 42m 30.8s time saved) :snowflake: 6 New Flaky :hourglass: 1 Performance Regression

:x: Failed Tests (874)

This report shows up to 5 failed tests.

test_200_request - test_aiohttp_client.py - Details

Expand for error

t request <Request GET /test/session/snapshot >:
  At snapshot (token='tests.contrib.aiohttp.test_aiohttp_client.test_200_request'):
   - Directory: /snapshots
   - CI mode: 1
   - Trace File: /snapshots/tests.contrib.aiohttp.test_aiohttp_client.test_200_request.json
   - Stats File: /snapshots/tests.contrib.aiohttp.test_aiohttp_client.test_200_request_tracestats.json
   At compare of 1 expected trace(s) to 1 received trace(s):
    At trace 'aiohttp.request' (2 spans):
     At snapshot compare of span 'aiohttp.request' at position 1 in trace:
      - Expected span:
..

test_200_request - test_aiohttp_client.py - Details

Expand for error

t request <Request GET /test/session/snapshot >:
  At snapshot (token='tests.contrib.aiohttp.test_aiohttp_client.test_200_request'):
   - Directory: /snapshots
   - CI mode: 1
   - Trace File: /snapshots/tests.contrib.aiohttp.test_aiohttp_client.test_200_request.json
   - Stats File: /snapshots/tests.contrib.aiohttp.test_aiohttp_client.test_200_request_tracestats.json
   At compare of 1 expected trace(s) to 1 received trace(s):
    At trace 'aiohttp.request' (2 spans):
     At snapshot compare of span 'aiohttp.request' at position 1 in trace:
      - Expected span:
..

test_200_request - test_aiohttp_client.py - Details

Expand for error

t request <Request GET /test/session/snapshot >:
  At snapshot (token='tests.contrib.aiohttp.test_aiohttp_client.test_200_request'):
   - Directory: /snapshots
   - CI mode: 1
   - Trace File: /snapshots/tests.contrib.aiohttp.test_aiohttp_client.test_200_request.json
   - Stats File: /snapshots/tests.contrib.aiohttp.test_aiohttp_client.test_200_request_tracestats.json
   At compare of 1 expected trace(s) to 1 received trace(s):
    At trace 'aiohttp.request' (2 spans):
     At snapshot compare of span 'aiohttp.request' at position 1 in trace:
      - Expected span:
..

test_200_request_post - test_aiohttp_client.py - Details

Expand for error

t request <Request GET /test/session/snapshot >:
  At snapshot (token='tests.contrib.aiohttp.test_aiohttp_client.test_200_request_post'):
   - Directory: /snapshots
   - CI mode: 1
   - Trace File: /snapshots/tests.contrib.aiohttp.test_aiohttp_client.test_200_request_post.json
   - Stats File: /snapshots/tests.contrib.aiohttp.test_aiohttp_client.test_200_request_post_tracestats.json
   At compare of 1 expected trace(s) to 1 received trace(s):
    At trace 'aiohttp.request' (2 spans):
     At snapshot compare of span 'aiohttp.request' at position 1 in trace:
      - Expected span:
..

test_200_request_post - test_aiohttp_client.py - Details

Expand for error

t request <Request GET /test/session/snapshot >:
  At snapshot (token='tests.contrib.aiohttp.test_aiohttp_client.test_200_request_post'):
   - Directory: /snapshots
   - CI mode: 1
   - Trace File: /snapshots/tests.contrib.aiohttp.test_aiohttp_client.test_200_request_post.json
   - Stats File: /snapshots/tests.contrib.aiohttp.test_aiohttp_client.test_200_request_post_tracestats.json
   At compare of 1 expected trace(s) to 1 received trace(s):
    At trace 'aiohttp.request' (2 spans):
     At snapshot compare of span 'aiohttp.request' at position 1 in trace:
      - Expected span:
..

New Flaky Tests (6)

test_schematization[schema_tuples0] - test_fastapi.py - Last Failure

Expand for error

t request <Request GET /test/session/snapshot >:
  At snapshot (token='tests.contrib.fastapi.test_fastapi.test_schematization[schema_tuples0]_9'):
   - Directory: /snapshots
   - CI mode: 1
   - Trace File: /snapshots/tests.contrib.fastapi.test_fastapi.test_schematization[schema_tuples0]_9.json
   - Stats File: /snapshots/tests.contrib.fastapi.test_fastapi.test_schematization[schema_tuples0]_9_tracestats.json
   At compare of 1 expected trace(s) to 1 received trace(s):
    At trace 'http.request' (4 spans):
     At snapshot compare of span 'http.request' at position 1 in trace:
      - Expected span:
..

test_schematization[schema_tuples1] - test_fastapi.py - Last Failure

Expand for error

t request <Request GET /test/session/snapshot >:
  At snapshot (token='tests.contrib.fastapi.test_fastapi.test_schematization[schema_tuples1]_9'):
   - Directory: /snapshots
   - CI mode: 1
   - Trace File: /snapshots/tests.contrib.fastapi.test_fastapi.test_schematization[schema_tuples1]_9.json
   - Stats File: /snapshots/tests.contrib.fastapi.test_fastapi.test_schematization[schema_tuples1]_9_tracestats.json
   At compare of 1 expected trace(s) to 1 received trace(s):
    At trace 'http.request' (4 spans):
     At snapshot compare of span 'http.request' at position 1 in trace:
      - Expected span:
..

test_schematization[schema_tuples2] - test_fastapi.py - Last Failure

Expand for error

t request <Request GET /test/session/snapshot >:
  At snapshot (token='tests.contrib.fastapi.test_fastapi.test_schematization[schema_tuples2]_9'):
   - Directory: /snapshots
   - CI mode: 1
   - Trace File: /snapshots/tests.contrib.fastapi.test_fastapi.test_schematization[schema_tuples2]_9.json
   - Stats File: /snapshots/tests.contrib.fastapi.test_fastapi.test_schematization[schema_tuples2]_9_tracestats.json
   At compare of 1 expected trace(s) to 1 received trace(s):
    At trace 'http.client.request' (4 spans):
     At snapshot compare of span 'http.client.request' at position 1 in trace:
      - Expected span:
..

test_schematization[schema_tuples3] - test_fastapi.py - Last Failure

Expand for error

t request <Request GET /test/session/snapshot >:
  At snapshot (token='tests.contrib.fastapi.test_fastapi.test_schematization[schema_tuples3]_9'):
   - Directory: /snapshots
   - CI mode: 1
   - Trace File: /snapshots/tests.contrib.fastapi.test_fastapi.test_schematization[schema_tuples3]_9.json
   - Stats File: /snapshots/tests.contrib.fastapi.test_fastapi.test_schematization[schema_tuples3]_9_tracestats.json
   At compare of 1 expected trace(s) to 1 received trace(s):
    At trace 'http.request' (4 spans):
     At snapshot compare of span 'http.request' at position 1 in trace:
      - Expected span:
..

test_schematization[schema_tuples4] - test_fastapi.py - Last Failure

Expand for error

t request <Request GET /test/session/snapshot >:
  At snapshot (token='tests.contrib.fastapi.test_fastapi.test_schematization[schema_tuples4]_9'):
   - Directory: /snapshots
   - CI mode: 1
   - Trace File: /snapshots/tests.contrib.fastapi.test_fastapi.test_schematization[schema_tuples4]_9.json
   - Stats File: /snapshots/tests.contrib.fastapi.test_fastapi.test_schematization[schema_tuples4]_9_tracestats.json
   At compare of 1 expected trace(s) to 1 received trace(s):
    At trace 'http.request' (4 spans):
     At snapshot compare of span 'http.request' at position 1 in trace:
      - Expected span:
..

:hourglass: Performance Regressions vs Default Branch (1)

test_commit_with_consume_single_message - test_kafka.py 1m 24.15s (+1m 21.06s, +2626%) - Details

May 24 '24 16:05 datadog-dd-trace-py-rkomorn[bot]

Datadog Report

Branch report: conti/inferred-service-autoinjection Commit report: 2872776 Test service: dd-trace-py

:x: 1800 Failed (14 Known Flaky), 161852 Passed, 1452 Skipped, 7h 19m 55.23s Total duration (3m 29.26s time saved) :snowflake: 65 New Flaky

:x: Failed Tests (1800)

This report shows up to 5 failed tests.

test_flask_ipblock_match_403[flask_appsec_good_rules_env] - test_appsec_flask_snapshot.py - Details

Expand for error

t request <Request GET /test/session/snapshot >:
  At snapshot (token='tests.contrib.flask.test_appsec_flask_snapshot.test_flask_ipblock_match_403[flask_appsec_good_rules_env]_220'):
   - Directory: /snapshots
   - CI mode: 0
   - Trace File: /snapshots/tests.contrib.flask.test_appsec_flask_snapshot.test_flask_ipblock_match_403[flask_appsec_good_rules_env]_220.json
   - Stats File: /snapshots/tests.contrib.flask.test_appsec_flask_snapshot.test_flask_ipblock_match_403[flask_appsec_good_rules_env]_220_tracestats.json
   At compare of 1 expected trace(s) to 1 received trace(s):
    At trace 'flask.request' (2 spans):
     At snapshot compare of span 'flask.request' at position 1 in trace:
      - Expected span:
..

test_flask_ipblock_match_403[flask_appsec_good_rules_env] - test_appsec_flask_snapshot.py

test_flask_ipblock_match_403[flask_appsec_good_rules_env] - test_appsec_flask_snapshot.py - Details

Expand for error

t request <Request GET /test/session/snapshot >:
  At snapshot (token='tests.contrib.flask.test_appsec_flask_snapshot.test_flask_ipblock_match_403[flask_appsec_good_rules_env]_220'):
   - Directory: /snapshots
   - CI mode: 0
   - Trace File: /snapshots/tests.contrib.flask.test_appsec_flask_snapshot.test_flask_ipblock_match_403[flask_appsec_good_rules_env]_220.json
   - Stats File: /snapshots/tests.contrib.flask.test_appsec_flask_snapshot.test_flask_ipblock_match_403[flask_appsec_good_rules_env]_220_tracestats.json
   At compare of 1 expected trace(s) to 1 received trace(s):
    At trace 'flask.request' (2 spans):
     At snapshot compare of span 'flask.request' at position 1 in trace:
      - Expected span:
..

test_flask_ipblock_match_403_json[flask_appsec_good_rules_env] - test_appsec_flask_snapshot.py

test_flask_ipblock_match_403_json[flask_appsec_good_rules_env] - test_appsec_flask_snapshot.py - Details

Expand for error

t request <Request GET /test/session/snapshot >:
  At snapshot (token='tests.contrib.flask.test_appsec_flask_snapshot.test_flask_ipblock_match_403_json[flask_appsec_good_rules_env]_220'):
   - Directory: /snapshots
   - CI mode: 0
   - Trace File: /snapshots/tests.contrib.flask.test_appsec_flask_snapshot.test_flask_ipblock_match_403_json[flask_appsec_good_rules_env]_220.json
   - Stats File: /snapshots/tests.contrib.flask.test_appsec_flask_snapshot.test_flask_ipblock_match_403_json[flask_appsec_good_rules_env]_220_tracestats.json
   At compare of 1 expected trace(s) to 1 received trace(s):
    At trace 'flask.request' (2 spans):
     At snapshot compare of span 'flask.request' at position 1 in trace:
      - Expected span:
..

New Flaky Tests (65)

test_custom_logging_injection - test_correlation_log_context.py - Last Failure

Expand for error

ssert {'env': '', '...7975066', ...} == {'env': '', '...7975066', ...}
 Omitting 4 identical items, use -vv to show
 Differing items:
 {'service': 'ddtrace'} != {'service': ''}
 Full diff:
   {
    'env': '',
 -  'service': '',
 +  'service': 'ddtrace',
 ?              +++++++
..

test_custom_logging_injection - test_correlation_log_context.py - Last Failure

Expand for error

ssert {'env': '', '...8496628', ...} == {'env': '', '...8496628', ...}
 Omitting 4 identical items, use -vv to show
 Differing items:
 {'service': 'ddtrace'} != {'service': ''}
 Full diff:
   {
    'env': '',
 -  'service': '',
 +  'service': 'ddtrace',
 ?              +++++++
..

test_get_log_correlation_context_no_active_span - test_correlation_log_context.py - Last Failure

Expand for error

ssert {'env': '', '...id': '0', ...} == {'env': '', '...id': '0', ...}
 Omitting 4 identical items, use -vv to show
 Differing items:
 {'service': 'ddtrace'} != {'service': ''}
 Full diff:
   {
    'env': '',
 -  'service': '',
 +  'service': 'ddtrace',
 ?              +++++++
..

test_get_log_correlation_context_no_active_span - test_correlation_log_context.py - Last Failure

Expand for error

ssert {'env': '', '...id': '0', ...} == {'env': '', '...id': '0', ...}
 Omitting 4 identical items, use -vv to show
 Differing items:
 {'service': 'ddtrace'} != {'service': ''}
 Full diff:
   {
    'env': '',
 -  'service': '',
 +  'service': 'ddtrace',
 ?              +++++++
..

test_child - test_middleware.py

Jun 20 '24 16:06 datadog-dd-trace-py-rkomorn[bot]

Benchmarks

Benchmark execution time: 2024-06-25 18:22:13

Comparing candidate commit 2872776e7ab0fcd9e65f99cd0e9f4698ba6ad685 in PR branch conti/inferred-service-autoinjection with baseline commit 053d891c725a4216aa51c33c70792a257cd96360 in branch main.

Found 2 performance improvements and 1 performance regressions! Performance is the same for 218 metrics, 9 unstable metrics.

scenario:otelspan-start

🟩 max_rss_usage [-7.488MB; -7.413MB] or [-16.367%; -16.203%]

scenario:span-add-metrics

🟩 max_rss_usage [-6.163MB; -6.091MB] or [-9.435%; -9.324%]

scenario:span-add-tags

🟥 max_rss_usage [+3.654MB; +4.277MB] or [+11.621%; +13.602%]

Jun 20 '24 16:06 pr-commenter[bot]

Questions:

Python internal services. Many python integrations use the component name as the service name, i.e. "aiohttp-web" unless the global is set. I have set these to use "aiohttp-web" if the inferred service is used (this causes base_service to be the running application)
Breaking changes. These changes will break anyone who has set alarms/etc on "unnamed-python-service". Is that ok?

Jun 24 '24 21:06 tabgok

This pull request has been automatically closed after a period of inactivity. After this much time, it will likely be easier to open a new pull request with the same changes than to update this one from the base branch. Please comment or reopen if you think this pull request was closed in error.

Sep 24 '24 00:09 github-actions[bot]