[Feature] Add Laravel Queue Job support

[NickStallman]

Is your feature request related to a problem? Please describe. The Laravel integration for DD Trace is good, however it completely omits Queue support. Any jobs that run are not traced at all.

Describe the solution you'd like Queue tracing can be accomplished with something along these lines:

\DDTrace\trace_method(
    'Illuminate\Queue\Worker',
    'process',
    function (\DDTrace\SpanData $span, array $args) {
        list($connectionName, $job, $options) = $args;

        if (!($job instanceof \Illuminate\Contracts\Queue\Job)) {
                return;
        }

        $payload = $job->payload();
        $span->resource = JobName::resolve($job->getName(), $payload);
        $span->name = 'laravel.queue.worker';
        $span->type = Type::MESSAGE_CONSUMER;
        $span->meta = [
                'waitTime' => !empty($payload['pushedAt']) ? (microtime(true) - $payload['pushedAt']) : false,
                'pushedAt' => !empty($payload['pushedAt']) ? date('c', $payload['pushedAt']) : false,
        ];
});

Additional context The aboove code has had some very basic tests and seems to work fine. Additional meta info might be handy but that's easy to add to.

I ran the queue worker like this:

DD_TRACE_CLI_ENABLED=true DD_TRACE_AUTO_FLUSH_ENABLED=true DD_TRACE_GENERATE_ROOT_SPAN=false ./artisan queue:work

[kalemdzievski]

Hey, @NickStallman I was researching about the very same thing, but no luck so far. I wanted to ask u, where did u place this code?

[NickStallman]

@kalemdzievski you can pop it anywhere really. a Laravel Service Provider is ideal for defining this in a boot() function.

[kalemdzievski]

Thank you!

[erictt]

How do you mark the failed jobs? It seems like there is no way to mark the span as error if the job failed.

[NickStallman]

@erictt Laravel Jobs are typically failed via Exceptions.

$job->hasFailed() can determine if it has failed or not. To get further information an extra hook on Job::fail($e) may be needed.

[nikalemdzievski]

Hey @NickStallman! Hope you are doing well. The code snipped you provided was really helpful, tho I noticed that timed-out jobs are not being traced. Any experience with this?

[NickStallman]

The timeout would probably be caught as an exception like above, however I haven't revisited my code since so I don't have an updated snippet sorry.

[dreanmer]

@NickStallman first, thanks for the snippet, work like a charm!

now I'm trying to ignore events that fails by some specific exceptions, do you have some tips to help? thanks!

[NickStallman]

I don't have an immediate solution off the top of my head for filtering the jobs that get reported. This hook just catches when the job is run, but to look at the result of the Job you'd need to hook the exception handler or something like that. I don't know how you'd prevent this job hook from firing if an exception occurred however.

[dreanmer]

I just found this documentation: https://docs.datadoghq.com/tracing/trace_collection/custom_instrumentation/php/?tab=currentspan#writing-the-custom-instrumentation

~the solution was easy, there is for someone who needs:~

          
            // the code above won't work
            \DDTrace\trace_method('Illuminate\Queue\Worker', 'process', function (\DDTrace\SpanData $span, array $args, $retval, $exception)
            {

                [...]

                // this was a mistake, the code above wont work, cuz process method captures the exception and don't let it explode, so we don't have access to it
                if ($exception !== null && $exception instanceof XxxxException) {
                    return;
                }

there is this last parameter, that carries the exception if it occurs!

also there is other (more complicated) ways to do that, but i will leave it here cuz can help someone: https://github.com/DataDog/dd-trace-php/issues/386

[NickStallman]

Oh well now that does indeed make it quite easy. :)

[dreanmer]

As the last code doesn't worked as expected, I've tried to change to trace the Jobs::fire method, so we can intercept the exceptions... but https://github.com/DataDog/dd-trace-php/issues/1921 and this is the case of this method... So I'm working on another option while this isn't fixed

[bwoebi]

Hey @NickStallman and everyone who long awaited this feature:

0.87.0 now finally features Laravel queue support.

Laravel queue processing traces are implemented as a distributed trace, i.e. it will be attached to the original trace scheduling the job.

The artisan job runner / horizon will contain placeholders for job processing. Currently this displays as a placeholder span without any further information in the datadog UI. In the near future we'll enable span links on our UI, which will connect these placeholders to the distributed trace, giving you visibility what jobs are running in specific artisan processes. I'll post an update here, once this is fully enabled. The logic is already present in the 0.87.0 tracer, just not generally available yet on the Datadog UI.

[morrisonlevi]

:wave: I left this comment in another thread and I'll copy it here in case it's valuable to you as well.

---

Just chiming in to add that v0.87.0 is out now!

There is also a new config mode in the datadog-setup.php script, which allows you to set service, env, version, etc, through the CLI. I think Laravel users might git a bit more out of this feature than some other users, so I'm sharing it with you:

# Install the tracer (no appsec, no profiling)
php datadog-setup.php --php-bin=php

# Get service and env from the .env file
ddservice=$(awk -F= '$1 ~ /^APP_NAME$/ {print $2}' < .env)
ddenv=$(awk -F= '$1 ~ /^APP_ENV$/ {print $2}' < .env)

# Grab version from git (or however you do it)
ddversion=$(git rev-parse HEAD)

# Sets service, env, and version through the installer.
# It uses INI syntax.
php datadog-setup.php config set --php-bin=php \
    -ddatadog.service="${ddservice?}" \
    -ddatadog.env="${ddenv?}" \
    -ddatadog.version="${ddversion?}"

Our online docs should be updated within the next week or so to reflect this.

Thanks for the patience on the Laravel queue worker support. We're excited to hear your feedback!

[NickStallman]

Excellent news! This looks much better than my interim hack so I'll be switching over. :)

[lifekent]

I've been struggling with custom trace recently, but it turned out that the Datadog doesn't work good with both env variables set in docker-compose.yml and php config. Once I moved all Datadog env variables to docker-compose.yml I started to see all dispatched jobs automatically without custom trace. I am using latest version 0.89.0

[jhjm32087]

I'm not seeing any Laravel Horizon on Datadog with 0.91.2. Are there any environmental settings to enable after the upgrade?

[PROFeNoM]

Hey @jhjm32087

This is definitely not normal :eyes: Could you please open an issue to follow this bug? I would first be particularly interested in knowing the following:

• The version of Laravel you are using
• Any exception debug output generated when using DD_TRACE_DEBUG=true
• Whether you still receive other traces from this service
• The output of phpinfo

[jhjm32087]

@PROFeNoM, I submitted a bug here https://github.com/DataDog/dd-trace-php/issues/2268

[PROFeNoM]

Hey everyone :wave:

Span Links are now broadly available as a Beta Feature in the UI and connect pushing and processing traces 😃

Please feel free to provide us with your feedback on this feature!