Unable to send events in CurlEventPublisher because of too many events

[umesecke]

Is this a support request? No

Describe the bug When too many flags are evaluated at once, the event processor queues up too many events to send them in one go. When the CurlEventPublisher then tries to send the events, a warning is created because the resulting curl CLI command is too long.

PHP Warning:  shell_exec(): Unable to execute '/usr/bin/env curl  -X POST --connect-timeout 1 -H 'Content-Type: application/json' -H 'Accept: application/json' -H 'Authorization: [...]' -H 'User-Agent: PHPClient/5.1.1' -H 'X-LaunchDarkly-Event-Schema: 2' -d '[ [...] ]' 'http://localhost:8030/bulk'>> /dev/null 2>&1 &' in /home/ubuntu/app/vendor/launchdarkly/server-sdk/src/LaunchDarkly/Impl/Integrations/CurlEventPublisher.php on line 104

In some places we can mitigate the problem by manually flushing the event queue but there are code paths where this is not practical without introducing lots of flush() calls everywhere. We could limit the capacity of the event queue down from 1000 to something like 50 but then we will loose events as well because once the queue hits capacity, more events seem to be discarded.

To reproduce

<?php

require_once __DIR__ . '/vendor/autoload.php';

$client = new \LaunchDarkly\LDClient("sdk-key", [
    'send_events' => true,
]);
$context = \LaunchDarkly\LDContext::create("user-key", "user");
for ($i = 0; $i < 1000; $i++) {
    $value = $client->variation("flag-key", $context);
}

// this should cause the warning
$client->flush();

Expected behavior

The expected behavior is that the event publisher sends the queued events in small (configurable) batches that fit into a single shell_exec call.

Logs see error message above

SDK version SDK version 5.1.1

Language version, developer tools PHP 8.1.23

OS/platform Ubuntu 22.04

Additional context

We are running a local relay-proxy in each host but this should not matter as the code above also reproduces the error message when bypassing the proxy.

[keelerm84]

Thank you for bringing this to our attention.

Instead of using the curl publisher, is the guzzle publisher an option for you?

The guzzle publisher is synchronous as opposed to the backgrounded curl command, so there is a small performance difference. However, since you are using a relay proxy, I would expect this impact to be quite small.

[umesecke]

I think in this specific case where we evaluate a flag with many different contexts inside a loop, we will rather reduce the queue capacity instead of switching to the Guzzle publisher.

Sending lots of small synchronous requests, even with small latency because of the local proxy, would probably add up. So in this case having only the first 20 or 50 events should be enough for the purpose of seeing the flag as in active use. In all other places the queue is flushed often enough for the small capacity not being a problem.

So yeah I think we can find a workaround for now but having a proper way for flushing lots of events at the same time would be really nice.