[opentelemetry-php-contrib] Doctrine connection spans not appearing in New Relic

[Ardenexal]

Describe your environment PHP 8.3.20

Composer

    "doctrine/doctrine-bundle": "^2.10",
    "doctrine/doctrine-migrations-bundle": "^3.2",
    "doctrine/orm": "^2.15",
    "open-telemetry/exporter-otlp": "^1.1.0",
    "open-telemetry/opentelemetry-auto-doctrine": "^0.2.0",
    "open-telemetry/opentelemetry-auto-psr3": "^0.1.0",
    "open-telemetry/opentelemetry-auto-psr14": "^0.0.4",
    "open-telemetry/opentelemetry-auto-psr16": "^0.0.4",
    "open-telemetry/opentelemetry-auto-psr18": "^1.1.0",
    "open-telemetry/opentelemetry-auto-symfony": "^1.0.0",

Steps to reproduce I've been trying out the new doctrine auto instrumentation and liking it so far but in New relic it not displaying as i would except.

The SELECT atc does not show in the database tab

But in the individual trace it does show correctly

What is the expected behavior? I should be able to see the Database statements in the database tab of New Relic.

What is the actual behavior? Currently they only seem to be shown in the Spans

Additional context

Looking a https://github.com/open-telemetry/opentelemetry-php-contrib/blob/main/src/Instrumentation/Doctrine/src/DoctrineInstrumentation.php I believe the issue is that the database attributes are not on the Connection::class spans

      ->setAttribute(TraceAttributes::SERVER_ADDRESS, AttributesResolver::get(TraceAttributes::SERVER_ADDRESS, func_get_args()))
                ->setAttribute(TraceAttributes::SERVER_PORT, AttributesResolver::get(TraceAttributes::SERVER_PORT, func_get_args()))
                ->setAttribute(TraceAttributes::DB_SYSTEM, AttributesResolver::get(TraceAttributes::DB_SYSTEM, func_get_args()))
                ->setAttribute(TraceAttributes::DB_NAMESPACE, AttributesResolver::get(TraceAttributes::DB_NAMESPACE, func_get_args()));

I'm guessing it's because there is some sort of Span link that is being used to link the connect span with the connection span but New Relic currently doesn't support link spans.

Could we add the following attributes to the Connection::class hooks db.system Additionally, I think we can add the following db.sql.table and db.operation for better UX in applications like New Relic.

Here is the documentation from New Relic if that's useful https://docs.newrelic.com/docs/opentelemetry/get-started/apm-monitoring/opentelemetry-apm-ui/#databases-page

[brettmc]

Just taking a look over doctrine auto-instrumentation, since I'm kicking the tires for my day job.

There isn't span linking, but that's the right way to do it. If NewRelic just needs one or more db.* semconv attributes to identify a span as being from a database, then that's easy enough, we can add db.operation.

I've addressed some of this in https://github.com/open-telemetry/opentelemetry-php-contrib/pull/374 - let's see if it is any nicer in NewRelic...

[Ardenexal]

Thank @brettmc, Had a look at the PR and it looks good and should fit my needs

[brettmc]

@Ardenexal these changes are in v0.3.0 - could you test it out and report back?

[Ardenexal]

@Ardenexal these changes are in v0.3.0 - could you test it out and report back?

I will test later today

[GrzegorzDrozd]

Please be aware that new relic does not support links between spans.

[Ardenexal]

@brettmc I am seeing the following error when getting an SQL error

<br />
<b>Warning</b>:  Doctrine\DBAL\Driver\PDO\Statement: :execute(): OpenTelemetry: post hook invalid signature, class=Doctrine\DBAL\Driver\PDO\Statement function=execute in <b>Unknown</b> on line <b>0</b><br />
<br />
<b>Warning</b>:  Doctrine\DBAL\Driver\Middleware\AbstractStatementMiddleware: :execute(): OpenTelemetry: post hook invalid signature, class=Doctrine\DBAL\Driver\Middleware\AbstractStatementMiddleware function=execute in <b>Unknown</b> on line <b>0</b><br />
<br />
<b>Warning</b>:  Doctrine\DBAL\Logging\Statement: :execute(): OpenTelemetry: post hook invalid signature, class=Doctrine\DBAL\Logging\Statement function=execute in <b>Unknown</b> on line <b>0</b><br />
<br />
<b>Warning</b>:  Doctrine\DBAL\Driver\Middleware\AbstractStatementMiddleware: :execute(): OpenTelemetry: post hook invalid signature, class=Doctrine\DBAL\Driver\Middleware\AbstractStatementMiddleware function=execute in <b>Unknown</b> on line <b>0</b><br />
<br />
<b>Warning</b>:  Symfony\Bridge\Doctrine\Middleware\Debug\DBAL3\Statement: :execute(): OpenTelemetry: post hook invalid signature, class=Symfony\Bridge\Doctrine\Middleware\Debug\DBAL3\Statement function=execute in <b>Unknown</b> on line <b>0</b><br />

it was generated off the following error An exception occurred while executing a query: SQLSTATE[42S02]: Base table or view not found: 1146 Table 'app.test_table' doesn't exist

[brettmc]

I might be missing something, but doctrine auto-instrumentation doesn't hook any method named execute that I can see. Which instrumentation packages do you have installed?

[Ardenexal]

This is what I currently have installed.

Could it be related to https://github.com/open-telemetry/opentelemetry-php-contrib/blob/main/src/Instrumentation/Doctrine/src/DoctrineInstrumentation.php#L167

open-telemetry/api                                          1.4.0              API for OpenTelemetry PHP.
open-telemetry/context                                      1.2.1              Context implementation for OpenTelemetry PHP.
open-telemetry/contrib-aws                                  1.0.0beta14        The Aws package for opentelemetry-php
open-telemetry/exporter-otlp                                1.3.2              OTLP exporter for OpenTelemetry.
open-telemetry/gen-otlp-protobuf                            1.5.0              PHP protobuf files for communication with OpenTelemetry OTLP collectors/servers.
open-telemetry/opentelemetry-auto-doctrine                  0.3.0              OpenTelemetry auto-instrumentation for Doctrine
open-telemetry/opentelemetry-auto-psr14                     0.0.4              OpenTelemetry auto-instrumentation for PSR-14 (Event Dispatcher).
open-telemetry/opentelemetry-auto-psr16                     0.0.4              OpenTelemetry auto-instrumentation for PSR-16 (Common Interface for Caching Libraries).
open-telemetry/opentelemetry-auto-psr18                     1.1.1              OpenTelemetry auto-instrumentation for PSR-18 (HTTP Client).
open-telemetry/opentelemetry-auto-psr3                      0.1.1              OpenTelemetry auto-instrumentation for PSR-3 (Logger Interface).
open-telemetry/opentelemetry-auto-symfony                   1.0.1              OpenTelemetry auto-instrumentation for Symfony
open-telemetry/sdk                                          1.6.0              SDK for OpenTelemetry PHP.
open-telemetry/sem-conv                                     1.32.1             Semantic conventions for OpenTelemetry PHP.

i think is issue is that ResultInterface $result can return null so it should be ?ResultInterface $result

  post: static function (\Doctrine\DBAL\Driver\Statement $statement, array $params, ResultInterface $result, ?Throwable $exception) {
                self::end($exception);
            }

[brettmc]

Sorry my mistake, my fork was not up-to-date. You're absolutely correct, and a PR to fix is submitted.

[Ardenexal]

I have had a look in New Relic after our deployment and still nothing in the database section. I think the issue is New Relic expects one span with the span name, and db.system, db.sql.table, and db.operation all on the same span.

Maybe we could do something like what was done for the PDO instrumentation to add the additional attributes to the execute and query spans when a env variarble is set like OTEL_PHP_INSTRUMENTATION_PDO_DISTRIBUTE_STATEMENT_TO_LINKED_SPANS https://github.com/open-telemetry/opentelemetry-php-contrib/blob/main/src/Instrumentation/PDO/src/PDOInstrumentation.php#L327C47-L327C112

[brettmc]

Maybe we could do something like what was done for the PDO instrumentation

At the moment, it feels like we're guessing about what might work with NewRelic - can I suggest you hack up the current doctrine package and inject the extra attributes you think are missing with some dummy values, and then verify if NR is happy? If it is, we'll have a good idea of next steps.

[Ardenexal]

Maybe we could do something like what was done for the PDO instrumentation

At the moment, it feels like we're guessing about what might work with NewRelic - can I suggest you hack up the current doctrine package and inject the extra attributes you think are missing with some dummy values, and then verify if NR is happy? If it is, we'll have a good idea of next steps.

Sure, I will try and find some time after work this week to have a tinker

[GrzegorzDrozd]

Hi @Ardenexal

I spoke with some people inside of newrelic and issue is that sem conv changed but they did not adjust their ingestion. So they still expect "old" names. Fix is in the works and should be deployed after July 21.

[Ardenexal]

Hi @Ardenexal

I spoke with some people inside of newrelic and issue is that sem conv changed but they did not adjust their ingestion. So they still expect "old" names. Fix is in the works and should be deployed after July 21.

Ah that's good to know. Hopefully the fix will be deployed soon

[Ardenexal]

New Relic released a change which has broken the implementation completely. They're how using metrics instead of spans to display the pages. I will need to use SpanMetrics i think to create metrics based on the Doctrine Spans. I'll look into this further.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.