Tracing with async/CLI setups

[SonarSoftware]

I'm currently using Swoole, but I'm guessing it's a similar issue for ReactPHP. Is there any direction on getting it working on an environment like this today? Or any plans to implement?

[labbati]

Hi @SonarSoftware, we definitely have plans for ReactPHP as it came up a few times now, and I guess a similar solution might apply to Swoole as well, as you are suggesting.

I cannot provide an ETA at the moment, though. Given that I am not expert about RactPHP/Swoole internals (I only can assume that there is some separate thread/event loop under the hood) this would make it difficult for you to support tracing without some work on our side due to the following reasons that comes to my mind (@SammyK and @morrisonlevi feel free to jump in as you might have more experience than me with those libraries):

• at the moment we flush traces on a register_shutdown_function. This would not work in your case as your script (I assume) runs as long as the app runs.
• Even if you manual invoke GlobalTracer::get()->flush() in some after-response hook that is provided by Swoole/ReactPHP, the Tracer remains in an unpredictable state as it approaches serving next request.
• On top of that, and this might be the really tricky part, We use at the moment one global tracer instance (as per OpenTracing approach) and span coming from different requests I am pretty sure would end up mixed in a giant basket.

Again, I have never used such libraries so I can only talk from the top of my head.

[SonarSoftware]

Yes, it is tricky. There are some ways around the global tracer issue - Swoole for example has a sandbox that is used to separate per-request items out. That being said, it's still not easy and it would definitely take some work.

It might be worth taking a look at https://github.com/upscalesoftware/swoole-newrelic for some inspiration - I've been poking at it a bit seeing if we can do something similar with ddtrace.

[nesl247]

Any update on this? I ask because we are in the process of migrating from NewRelic and ran into this. It is not an issue we expected to show up.

[morrisonlevi]

I am not aware of any updates specific to ReactPHP or Swoole, but I know some of the underlying engineering challenges that assume a single global trace are being worked on.

One thing to set expectations on is what in these products gets traced. To that end you could help by sharing what features you are using. As an example, I found a React\Http\Server class -- I assume that users would want each new request that the server processes to get traced independently. Telling us which pieces of Swoole and ReactPHP are being used can help us know where to look in docs, examples, etc when this work gets prioritized.

Another thing worth knowing is what PHP version and which version of Swoole or ReactPHP you are using, to make sure our work is actually relevant to what our users need.

[nesl247]

I believe we're using Swoole v4.4.15 at this time with PHP 7.4. And yes, we would expect each new request that swoole processes to be traced independently.

Also, thanks for the fast update.

[SonarSoftware]

Yep, exactly. Just treat each request as independent. I think this might be fairly easy to do by hooking into the onRequest method.. I'm planning to take a look at this myself soon.

[mrtus]

HI @morrisonlevi @labbati Could you provide us with an update on the work being done to make this possible?

And would it maybe be possible to elaborate a little bit more on this:

Even if you manual invoke GlobalTracer::get()->flush() in some after-response hook ..., the Tracer remains in an unpredictable state as it approaches serving next request.

As I'm currently looking into getting our distributed services through queueing connected to each other with tracing (we already have our logs connected through the same way with a trace_id property). I would, just like it is already done with curl, attach the tracing data on the message when published and get that data from the message when consumed by a long running process worker. After this message is consumed, I would of course want to call a flush, but I'm reading here that the tracer would be in an unpredictable state. Can you elaborate what you mean with unpredictable, what are the side-effects of this, and would there be some sort of remenand traces that would tie multiple messages together?

[anderson-xd]

Hi, any updates about this feature?

Thanks!

[labbati]

Hi @anderson-flores! At the moment we were not able to schedule the work for this and we cannot commit to a specific timeline. We will update as soon as possible.

[wregis]

I guess I've got a similar problem with Laravel Horizon running for days on a server. It would be nice to trace jobs individually.

[alexjose]

Swoole and ReactPHP is gaining more and more traction. At the same time, we have Laravel Octane working with Swoole under the hood. It would be very great to speed up the support of the coroutines in Datadog.

[anderson-xd]

Hello, any updates?

A year has passed and more and more swoole is gaining attention from the community, especially after the release of Laravel Octane and other packages with Swoole for Laravel, do we have any estimate when this will be implemented? Or if it won't be

thks.

[bwoebi]

We do not yet have concrete updates on this, except that it is on our roadmap to have at least the possibility of manual tracing of async applications by now (we expect some actual results regarding async surfacing this year) and new APIs are being designed with async in mind (e.g. the usage of a span stack like we currently have is inherently incompatible with async use cases).

Concrete new integrations will be evaluated once support for async instrumenting has landed at all.

[elijahchancey]

We'd really really love to have this!

[chrisbradleydev]

Any news on tracing for Laravel Octane Swoole?

[bwoebi]

I sadly have to repeat what I said in February. Hopefully basic fiber support, including simple async support will land within a couple months. Only then, we will be able to look into laravel octane or swoole.

[chrisbradleydev]

@bwoebi, thanks for the quick reply.

[egonbraun]

Quick question, is this support with RoadRunner instead of Swoole? Or is this not support due to the async nature of Laravel?

[donparapidos]

@bwoebi any news on this, in few weeks it will be 3 years since initial question ? quite some time

[morrisonlevi]

In ddtrace 0.82.0, we added a Roadrunner integration. Bob will be unavailable until the New Year, and I am filling in as best as I can. I don't know what's on the roadmap, or what's blocking more Async/CLI setups, sorry!

[githubeto]

I set up ddtrace 0.82.0 in the Laravel Octane (Roadrunner) application, but still no trace data was output to APM. Does that mean that 0.82.0 is still not supported? And also, when do you plan to release?

[bwoebi]

Hey @githubeto, Minimal tracing data about the requests occurring should appear since 0.82.0 with Laravel Octane running on roadrunner. I'd like to ask whether you did enable CLI? Roadrunner is spawning CLI processes, which needs to be explicitly enabled in ini via datadog.trace.cli_enabled=1 (or DD_TRACE_CLI_ENABLED=1 env var).

[githubeto]

@bwoebi , appreciate your quick reply. I had already set DD_TRACE_CLI_ENABLED=true in my Docker laravel application. Extract a part of the dockerfile.

• dockerfile (laravel application)

RUN curl -LO https://github.com/DataDog/dd-trace-php/releases/latest/download/datadog-setup.php
RUN php datadog-setup.php --php-bin=all

ENV DD_TRACE_CLI_ENABLED true
ENV DD_TRACE_GENERATE_ROOT_SPAN false
ENV DD_TRACE_AUTO_FLUSH_ENABLED true
ENV DD_TRACE_GENERATE_ROOT_SPAN true
ENV DD_TRACE_ANALYTICS_ENABLED true
ENV DD_TRACE_LARAVEL_ENABLED true

This is an application running on AWS Fargate, and the task definition also includes the following settings.

• laravel application container

                {
                    "name": "DD_ENV",
                    "value": "infra"
                },
                {
                    "name": "DD_SERVICE",
                    "value": "api"
                },
                {
                    "name": "DD_VERSION",
                    "value": "beta"
                },

• datadog agent container

               {
                    "name": "DD_SITE",
                    "value": "datadoghq.com"
                },
                {
                    "name": "DD_PROCFS_PATH",
                    "value": "/proc"
                },
                {
                    "name": "DD_APM_NON_LOCAL_TRAFFIC",
                    "value": "true"
                },
                {
                    "name": "DD_APM_ENABLED",
                    "value": "true"
                },
                {
                    "name": "ECS_FARGATE",
                    "value": "true"
                },
                {
                    "name": "DD_LOGS_ENABLED",
                    "value": "true"
                },
                {
                    "name": "DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL",
                    "value": "true"
                },
                {
                    "name": "DD_CONTAINER_EXCLUDE",
                    "value": "name:datadog-agent"
                },
                {
                    "name": "DD_PROCESS_AGENT_ENABLED",
                    "value": "true"
                },
                {
                    "name": "DD_APM_ENV",
                    "value": "infra"
                }

~~~

            "secrets": [
                {
                    "name": "DD_API_KEY",
                    "valueFrom": "xxxxxxxxxxxxx"
                }
            ],

If I change the application server configuration to nginx + php-fpm instead of laravel octane(roadrunner), tracing data about the requests occurring appear, so I think the settings I wrote above are fine.

[donparapidos]

4 months passed since I asked and over 3 years since this issue was raised and we still got no answer about the roadmap. Will Datadog support apm on Swoole, ever ? Swoole is now super popular and I cant believe company like Datadog ignores it

[bwoebi]

Hey @donparapidos,

We are sorry for not supporting Swoole yet.

However, your comment just prompted me into looking again into the swoole project. They seem to have committed support for the Zend Fiber API in https://github.com/swoole/swoole-src/commit/a941e94497af6326c72c70e62a031d3efc8c6d3b, which was released in February in v5.0.2.

Given that this support exists now, we are now able to put Swoole support with a higher priority on our roadmap!

Thanks for bringing it to my attention again. :-)

[bwoebi]

I haven't tested it, but I think just judging from that commit, that ddtrace should run on v5.0.2 without crashing (proper instrumentation is still missing, but there's a chance it will run at all).

[s1rc]

With the release of Horizon async queue support in 0.87.0, any chances we will see proper support for Octane soon?

[leonvd6]

Any updates on Swoole support? We migrating from PHP-FPM to OpenSwoole with Laravel Octane and the performance gains are massive but then we lose Datadog support.

[bwoebi]

OpenSwoole does not provide an API for tracking context switches. (Development seems also to be relatively quiet there.)

With Swoole, we're targeting next quarter to have a closer investigation and possibly actual support.

[carlosevieira]

Any updates on this @bwoebi ? I just migrated from php-fpm to Swoole, but I'm just missing datadog :/