Errors/Warnings in service's log

[guyzmo]

When I look at the logs in google cloud (like for the deployments), I have many warnings like:

/usr/local/lib/python3.12/site-packages/ddtrace/vendor/psutil/_pslinux.py:527: RuntimeWarning: 'sin' and 'sout' swap memory stats couldn't be determined and were set to 0 ([Errno 2] No such file or directory: '/proc/vmstat'

or like:

2024-10-09 17:18:04,601 WARNING [ddtrace.internal.writer.writer] [writer.py:255] [dd.service=core dd.env=staging dd.version=1 dd.trace_id=0 dd.span_id=0] - sent 2.79KB in 5.23711s to http://localhost:8126/v0.5/traces

or

failed to submit span stats to the Datadog agent at http://localhost:8126/v0.6/stats, 2 additional messages skipped

Is there a way to silence them? Or maybe I have something wrong that I should fix? Or maybe there's some env variables I should turn off (for swap memory stats)?

[hulkholden]

In our logger setup we do this to silence these chatty messages:

    logging.getLogger("ddtrace.internal.writer").setLevel(logging.FATAL)

(I don't know if it's the best way to handle them but it works for us).

[wantsui]

@guyzmo - Does the application use logging in general?

If so, by default dd-trace-py respects the logging level configured for it, so what @hulkholden is suggesting would be the best path!

More details can be found in Scenario 1: ddtrace version 2.x and higher .

To apply this to all of the logs emitted by dd-trace-py instead of only the internal writer, you can also just target ddtrace, ie:

logging.getLogger("ddtrace").setLevel(logging.WARNING)

However, if you ever do want to us to debug/troubleshoot the various error messages, like the ones involving failing to send stats, then we'd need the ddtrace in debug mode or higher.

---

Regarding your sample logs:

Example 1:

/usr/local/lib/python3.12/site-packages/ddtrace/vendor/psutil/_pslinux.py:527: RuntimeWarning: 'sin' and 'sout' swap memory stats couldn't be determined and were set to 0 ([Errno 2] No such file or directory: '/proc/vmstat'

There are a few causes of this happening, so if you open a support ticket with us and tell us more about the environment and google cloud deployment, we may be able to provide some suggestions.

Example 2:

2024-10-09 17:18:04,601 WARNING [ddtrace.internal.writer.writer] [writer.py:255] [dd.service=core dd.env=staging dd.version=1 dd.trace_id=0 dd.span_id=0] - sent 2.79KB in 5.23711s to http://localhost:8126/v0.5/traces

This message isn't an error but just the Python tracer saying it submitted data to the Trace Agent in the Datadog Agent

Example 3:

failed to submit span stats to the Datadog agent at http://localhost:8126/v0.6/stats, 2 additional messages skipped

This indicates an issue sending data to the Datadog Agent, but if this occurred before the log about sending traces, it might be okay depending on when your Datadog Agent in this set up is ready to collect the traces. It's hard to confirm the impact without knowing more of the frequency. (If it's a concern, a support ticket would be the best path forward so we can fully study the logs).

[guyzmo]

FWIW, I believe that a lib/an agent should output warning level logs for missing/bad configurations that are recoverable, error level logs for issues that are leading to error or lost information, and everything else should be INFO or less.

So setting the logger to ignore anything less than FATAL seems extreme, and makes me afraid of silencing logs that could help me improve my configuration.

But if that's the recommended way, I'm doing it.

Also, I did open a ticket for other issues I had in my logs (actual errors), and I figured those were more like an annoyance, not an actual issue, so it's better fitted here ☺

About the ①

I'm deploying on google cloud run, and obviously the instances don't implement the /proc/vmstat so there could be some sort of env/test to determine that there's no point in checking the swap memory stats.

About the ②

Why would that be a warning if it's just informational? That one should be INFO level at most! 😉

About the ③

Well, either there's a misconfiguration/error and I'm actually missing recorded traces not being pushed to the APM service, and this error is an indicator and I need to fix my config (and my question would be how?), or this is not an error, it shouldn't appear in my logs or be at most INFO level ☺

[wantsui]

Hey @guyzmo - thanks for the feedback! I didn't forget about this issue - I'm going to mark this with a feature request label since it would require some long term changes to how we approach logging levels as I saw a few other threads about logging as well and if/when we approach this, we'd need to consider all the use cases. I agree the current logging level behavior could be improved though!

(and my question would be how?)

This is tricky for sporadic connection losses since it gets into the weeds of investigating network configurations. If you find yourself seeing this issue a lot, a support ticket would be a good start since I'm afraid I don't have any easy suggestions for fixing sporadic drops, sorry about that!

[github-actions[bot]]

This issue has been automatically closed after a period of inactivity. If it's a feature request, it has been added to the maintainers' internal backlog and will be included in an upcoming round of feature prioritization. Please comment or reopen if you think this issue was closed in error.