DataDog
[USM] Add support for blocked process in USM status and debug endpoint
What does this PR do?
Adding support for blocked process in USM status and debug endpoint.
The function GetBlockedPathIDsList returns a list of blocked file identifiers for a specified type of process.
These identifiers are unique based on their device and inode numbers, ensuring that the blocked files can be accurately identified and managed within the context of the specified process.
Motivation
Improve USM debugging tools.
Additional Notes
To do:
- [ ] Should validate the changes in staging to ensure there is no negative impact on memory consumption.
QA steps: Enable system-probe with TLS monitoring:
service_monitoring_config:
enabled: true
tls:
go:
enabled: true
native:
enabled: true
enable_http_monitoring: true
Run the agent
Run agent status command
Expect to see
============
System Probe
============
Status: Running
Uptime: 250ns
Last Updated: 2024-07-01 05:34:00 PDT / 2024-07-01 12:34:00 UTC (1719837240000)
USM
===
Status: running
Last Check: 2024-07-01 05:33:57 PDT / 2024-07-01 12:33:57 UTC (1719837237000)
Traced Programs: [
{
"FilePath": "/usr/lib/aarch64-linux-gnu/libgnutls.so.30.31.0",
"PIDs": [
320208,
724
],
"ProgramType": "shared_libraries"
},
{
"FilePath": "/usr/lib/aarch64-linux-gnu/libcrypto.so.3",
"PIDs": [
679,
815,
303952,
320208,
326754,
348721,
349182,
349657,
],
"ProgramType": "shared_libraries"
},
{
"FilePath": "/usr/bin/dockerd",
"PIDs": [
974
],
"ProgramType": "go-tls"
}
]
Blocked Processes: [
{
"PathIdentifiers": [],
"ProgramType": "shared_libraries"
},
{
"PathIdentifiers": [
{
"Dev": 64768,
"Inode": 398470,
"SamplePath": "/proc/1/root/usr/lib/systemd/systemd"
},
{
"Dev": 64768,
"Inode": 398484,
"SamplePath": "/proc/443/root/usr/lib/systemd/systemd-journald"
},
{
"Dev": 64768,
"Inode": 407458,
"SamplePath": "/proc/482/root/usr/sbin/multipathd"
},
{
"Dev": 64768,
"Inode": 405991,
"SamplePath": "/proc/485/root/usr/bin/udevadm"
},
{
"Dev": 64768,
"Inode": 397805,
"SamplePath": "/proc/679/root/usr/lib/systemd/systemd-timesyncd"
},
{
"Dev": 64768,
"Inode": 398490,
"SamplePath": "/proc/722/root/usr/lib/systemd/systemd-networkd"
},
{
"Dev": 64768,
"Inode": 393913,
"SamplePath": "/proc/740/root/usr/bin/dbus-daemon"
},
{
"Dev": 64768,
"Inode": 407417,
"SamplePath": "/proc/744/root/usr/sbin/irqbalance"
},
{
"Dev": 64768,
"Inode": 393338,
"SamplePath": "/proc/746/root/usr/bin/python3.10"
},
{
"Dev": 64768,
"Inode": 407001,
"SamplePath": "/proc/747/root/usr/libexec/polkitd"
},
{
"Dev": 64768,
"Inode": 398497,
"SamplePath": "/proc/724/root/usr/lib/systemd/systemd-resolved"
},
{
"Dev": 64768,
"Inode": 398486,
"SamplePath": "/proc/757/root/usr/lib/systemd/systemd-logind"
},
{
"Dev": 64768,
"Inode": 407026,
"SamplePath": "/proc/763/root/usr/libexec/udisks2/udisksd"
},
{
"Dev": 64768,
"Inode": 394064,
"SamplePath": "/proc/765/root/usr/bin/containerd"
},
{
"Dev": 64768,
"Inode": 407488,
"SamplePath": "/proc/749/root/usr/sbin/rsyslogd"
},
{
"Dev": 64768,
"Inode": 407334,
"SamplePath": "/proc/811/root/usr/sbin/agetty"
},
{
"Dev": 64768,
"Inode": 407500,
"SamplePath": "/proc/815/root/usr/sbin/sshd"
},
{
"Dev": 64768,
"Inode": 436820,
"SamplePath": "/proc/863/root/usr/bin/prltoolsd"
},
{
"Dev": 64768,
"Inode": 436816,
"SamplePath": "/proc/877/root/usr/bin/prltimesync"
},
{
"Dev": 1796,
"Inode": 568,
"SamplePath": "/proc/754/root/snap/snapd/21761/usr/lib/snapd/snapd"
},
{
"Dev": 64768,
"Inode": 407355,
"SamplePath": "/proc/777/root/usr/sbin/cron"
},
{
"Dev": 64768,
"Inode": 410335,
"SamplePath": "/proc/791/root/usr/sbin/ModemManager"
},
{
"Dev": 64768,
"Inode": 393825,
"SamplePath": "/proc/327528/root/usr/bin/bash"
},
{
"Dev": 64768,
"Inode": 406997,
"SamplePath": "/proc/320208/root/usr/libexec/packagekitd"
},
{
"Dev": 64768,
"Inode": 399478,
"SamplePath": "/proc/351251/root/usr/bin/sudo"
},
{
"Dev": 64768,
"Inode": 1049591,
"SamplePath": "/proc/352180/root/home/vagrant/go/bin/dlv"
},
{
"Dev": 64768,
"Inode": 393995,
"SamplePath": "/proc/352212/root/usr/bin/getconf"
}
],
"ProgramType": "go-tls"
}
]
NPM
===
Status: Running
Last Check: 2024-07-01 05:33:54 PDT / 2024-07-01 12:33:54 UTC (1719837234000)
Event Monitor
================
Status: Running
Process
=======
Status: Running
Example with non blocked results:
universal_service_monitoring:
blocked_process: null
state: disabled
Possible Drawbacks / Trade-offs
Describe how to test/QA your changes
Codecov Report
Attention: Patch coverage is 0% with 1 line in your changes missing coverage. Please review.
Project coverage is 25.31%. Comparing base (
95cd819) to head (bbbbd88).
:exclamation: Current head bbbbd88 differs from pull request most recent head d23cb84
Please upload reports for the commit d23cb84 to get more accurate results.
| Files | Patch % | Lines |
|---|---|---|
| cmd/system-probe/modules/network_tracer.go | 0.00% | 1 Missing :warning: |
Additional details and impacted files
@@ Coverage Diff @@
## main #26363 +/- ##
===========================================
- Coverage 44.92% 25.31% -19.62%
===========================================
Files 2358 31 -2327
Lines 273821 1675 -272146
===========================================
- Hits 123025 424 -122601
+ Misses 141097 1198 -139899
+ Partials 9699 53 -9646
| Flag | Coverage Δ | |
|---|---|---|
| amzn_aarch64 | 25.34% <0.00%> (-20.43%) |
:arrow_down: |
| centos_x86_64 | 25.34% <0.00%> (-20.34%) |
:arrow_down: |
| ubuntu_aarch64 | 25.34% <0.00%> (-20.43%) |
:arrow_down: |
| ubuntu_x86_64 | 25.34% <0.00%> (-20.43%) |
:arrow_down: |
| windows_amd64 | ? |
Flags with carried forward coverage won't be shown. Click here to find out more.
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
![codecov[bot] avatar](https://avatars.githubusercontent.com/in/254?v=4 "Published by a pub.dev verified publisher"){kind=small}
that's not PIDs
- PIDs:
- Dev: 64768
Inode: 398484
- Dev: 64768
Inode: 407458
- Dev: 64768
Inode: 405991
- Dev: 64768
Inode: 397805
- Dev: 64768
Inode: 398497
- Dev: 64768
Inode: 393913
- Dev: 64768
Inode: 407417
- Dev: 64768
Inode: 393338
- Dev: 64768
Inode: 407001
- Dev: 64768
Inode: 398490
- Dev: 1796
Inode: 568
- Dev: 64768
Inode: 407026
- Dev: 64768
Inode: 407488
- Dev: 64768
Inode: 398486
- Dev: 64768
Inode: 394064
- Dev: 64768
Inode: 410335
- Dev: 64768
Inode: 407334
- Dev: 64768
Inode: 436820
- Dev: 64768
Inode: 436816
- Dev: 64768
Inode: 407355
- Dev: 64768
Inode: 398470
- Dev: 64768
Inode: 393825
- Dev: 64768
Inode: 407500
- Dev: 64768
Inode: 1.049591e+06
- Dev: 64768
Inode: 399478
- Dev: 64768
Inode: 393995
Test changes on VM
Use this command from test-infra-definitions to manually test this PR changes on a VM:
inv create-vm --pipeline-id=38242584 --os-family=ubuntu
Note: This applies to commit 9be59973
![pr-commenter[bot] avatar](https://avatars.githubusercontent.com/u/365230?v=4 "Published by a pub.dev verified publisher"){kind=small}
Regression Detector
Regression Detector Results
Run ID: 82f21968-f2dd-4c9d-b966-4caa8a893904 Metrics dashboard Target profiles
Baseline: 7ad93573342f9c0b6526ee6ef4453325e28939ef Comparison: dee35a39f9eb999c81eedca183ad42ee35246115
Performance changes are noted in the perf column of each table:
- ✅ = significantly better comparison variant performance
- ❌ = significantly worse comparison variant performance
- ➖ = no significant change in performance
No significant changes in experiment optimization goals
Confidence level: 90.00% Effect size tolerance: |Δ mean %| ≥ 5.00%
There were no significant changes in experiment optimization goals at this confidence level and effect size tolerance.
Fine details of change detection per experiment
| perf | experiment | goal | Δ mean % | Δ mean % CI | links |
|---|---|---|---|---|---|
| ➖ | tcp_syslog_to_blackhole | ingress throughput | +4.13 | [-8.98, +17.25] | Logs |
| ➖ | basic_py_check | % cpu utilization | +0.97 | [-1.68, +3.62] | Logs |
| ➖ | idle | memory utilization | +0.50 | [+0.47, +0.53] | Logs |
| ➖ | otel_to_otel_logs | ingress throughput | +0.06 | [-0.75, +0.87] | Logs |
| ➖ | tcp_dd_logs_filter_exclude | ingress throughput | -0.00 | [-0.01, +0.01] | Logs |
| ➖ | uds_dogstatsd_to_api | ingress throughput | -0.00 | [-0.00, +0.00] | Logs |
| ➖ | file_tree | memory utilization | -0.29 | [-0.34, -0.24] | Logs |
| ➖ | uds_dogstatsd_to_api_cpu | % cpu utilization | -0.45 | [-1.32, +0.42] | Logs |
| ➖ | pycheck_1000_100byte_tags | % cpu utilization | -0.74 | [-5.55, +4.07] | Logs |
Explanation
A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".
For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:
-
Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.
-
Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.
-
Its configuration does not mark it "erratic".
:steam_locomotive: MergeQueue: pull request added to the queue
The median merge time in main is 25m.
Use /merge -c to cancel this operation!