Add ability to automatically break out service names based on integration

[brettlangdon]

Description/comment taken from @goodspark on #1044

---

Thanks for opening this, but I think this is slightly different from my last comment (https://github.com/DataDog/dd-trace-py/issues/547#issuecomment-462074146).

I'd still like some way to have a better break out of service names based on actual resource URLs. Or something more specific than just "postgres" or "memcached". Imagine how in 'prod' I might have 50 different services, lets say 10 of them are using their own postgres DB instances/clusters.

If you were to look at the service map for such a layout, what would it look like? My guess is you'd see 10 edges to a 'postgres' node. But all 10 of those DB instances/clusters are independent of each other, right? The same argument for memcached, lambda, etc, etc. I haven't actually tested this for DBs, but I already see similar side effects for 'grpc' "mega-nodes" in the service map. Example:

When in reality, a call to grpc-based service A has nothing to do with grpc-based service B. All my grpc calls don't go through some single grpc routing service that I'm running.

Right now to get better break outs of services, I'm having to reach around and use an undocumented-but-public SQLAlchemy tracing function (trace_engine specifically, and undocumented as as of https://github.com/DataDog/dd-trace-py/pull/261) and manually set the service name, based on the DB URL and DB name. I don't have an idea on how to do this for memcached clusters/instances, lambdas, etc yet.

As a point-of-reference, the Python requests patching has an option to split outgoing traced requests by domain name (the split_by_domain option). This is more what I'm looking for, although it's not so easy to define for things like DBs, cache clusters, etc (eg. How do you uniquely name SQLite in-memory DBs used by different services? What about cache clusters that are effectively load balanced on the client-side via consistent hashing mechanisms? Should those all be different "cache" services, each with their unique endpoint? My gut reaction is no, but then that makes service naming even trickier).

Perhaps another question is in order: How does Datadog recommend we name services to get the most benefit out of all of DD's services (APM, service maps, trace analytics, logging, monitoring, watchdogs, etc)?

If you look at https://docs.datadoghq.com/tracing/visualization/#services, notice how even the documentation is using a service name that isn't normally possible using ddtrace's latest implementation (I'm referring to the quote, "The screenshot below is a microservice distributed system for an e-commerce site builder. There’s a web-store, ad-server, payment-db, and auth-service all represented as services in APM.") because you'd just end up with something like "postgres" or "mysql" or "sqlite", not "payment-db".

[bparker98]

This is exceedingly annoying with the psycopg integration as there doesn't seem to be any way to override the service name. It makes handling multiple databases very difficult since the UI has very poor filtering in the service view.

[Kyle-Verhoog]

:wave: @bparker98 we have a method to override the service name for particular instances outlined here: http://pypi.datadoghq.com/trace/docs/db_integrations.html#module-ddtrace.contrib.psycopg

Let me know if you have any problems getting that working!

[bparker98]

This doesn't help because we don't create our own psycopg database connections. It's all done by Django. It should be able to be overridden on a global level.

[Kyle-Verhoog]

Oh yes, unfortunately we currently don't have a method to override service names of db connections in Django. This is something we're working on.

It's a bit tricky to do it on a global level however. We'd need some kind of unique property of a connection to correlate a service name to (a database URL might be such a property). Something like:

ddtrace.config.psycopg.service_mapping= [("localhost:5432", "db-service-name-1"), ("localhost:5433", "db-service-name-2")]

which would have a corresponding environment variable.

What do you think? Did you have a particular idea of what this might look like at the global level?

[bparker98]

Generally our Django apps only have one database connection, we would just like to be able to rename that so that they all don't get lumped under one Django service in APM. It doesn't need to be any more complicated than that for our use case.

[Kyle-Verhoog]

@bparker98 so we do have the option (kind of) for doing it for Django. http://pypi.datadoghq.com/trace/docs/web_integrations.html#django

[Kyle-Verhoog]

There is still an issue where the underlying database library might be instrumented separately and that will have a different service name. This is something we're looking to fix in v0.42: both providing the ability to customize the service name for the database library and/or for the library integration to inherit the service name from the Django span)

[stephledev]

So what is the current state of things? The NodeJs client supports this behaviour for years, while there doesn't seem to be any movement around this topic on the python side. It's quite awkward using instrumentation clients that behave differently depending on the runtime, it especially leads to confusion in the service overview.

[brettlangdon]

I am going to close this issue since there are some changes planned for service naming (cannot provide an ETA at this time) which will improve the experience overall, but would not be compatible with this request.

In the meantime, using https://ddtrace.readthedocs.io/en/stable/configuration.html#DD_SERVICE_MAPPING is probably the best option.

[fajpunk]

Any updates on the service naming changes? We're running into this same problem for multiple applications.