Unserializable closures

[Freeaqingme]

I was just experimenting a little bit with ext-fiber, and was wondering whether running closures concurrently will be supported. Say I have an array of closures that will fetch stuff from mysql and then post it to a - slow - SOAP api, will it be possible to wrap that in a fiber and have them execute concurrently?

I tried to use the existing amphp tools (on v2.x-dev), but noticed there's no amphp/functions yet for 2.x, and amphp/parallel still attempts to serialize it.

[kelunik]

Yes, everything that currently works will continue working.

However, it'll be better to use a non-blocking implementation like amphp/mysql + amphp/http-client instead of amphp/parallel, since your work load is IO bound.

[Sarke]

@Freeaqingme The issue you're having with parallel is that the work is done in a different process or thread, so any data that needs to be shared needs to be serialized first, and then unserialized in the other process/thread. This is something PHP can't do natively, hence the errors.

This fiber extension is run in the same thread so there is no need for this, and you shouldn't have the same issue.

Note that you can try to use https://github.com/opis/closure to serialize the closures if you still want to go the parallel route.

[Freeaqingme]

@kelunik thanks for replying. I realize that what's already working will keep working. But my point was that with the current (stable) code, it doesn't work because the closures are unserializable. I'm hoping that with ext-fiber there's no need any more to serialize closures, and they can be executed in concurrently. But I'm not sure if that's the case?

@Sarke Thanks, but I don't think my closures could be serialized at all, and even if they could I'm not looking for them to be, I think for complex closures that capture large swaths of an application that'd be to brittle/fragile. I'd much rather just have 'proper' coroutines that don't need any serialization, that's why I was playing with ext-fiber :)

[kelunik]

They can be executed concurrently without serialization, but they can't be executed in parallel. You'll need a non-blocking IO implementation.

[trowski]

@Freeaqingme It seems a bit like you're confusing fibers with threads. Take a look at krakjoe/parallel and the v2 branch amphp/parallel. Don't try to serialize your code, write a Amp\Parallel\Worker\Task implementation and send only the necessary data instead.

[Gemorroj]

In fact its a big problem. For example: i cant parallel code on stable version of amp https://github.com/Gemorroj/parallel-nonserializable and cant parallel on v2 version https://github.com/Gemorroj/parallel-nonserializable/tree/fiber symfony/http-client - it is only example. in real code it contains doctrine and a lot of another non serializable code.

[Sarke]

@Gemorroj that is using parallel workers and not fibers. Your issue belongs in https://github.com/amphp/parallel

[faizanakram99]

fwiw, ext-mysqli supports async since 2009 or so, see https://www.php.net/manual/en/mysqli.poll.php

$connections = [];
$connections[] = $connection = new \mysqli(...$params);
$connection->query($sql, \MYSQLI_ASYNC);

// optionally more connection objects can be added to connection array, 
// the connections can be created with different mysql servers too.
// $connections[] = $conn = new \mysqli(...$otherParams); 
// $conn->query($anotherQuery, \MYSQLI_ASYNC);

do {
    $read = $error = $reject = $connections;
    $count = \mysqli::poll($read, $error, $reject, 1);

    if (0 < $count) {
        foreach ($read as $connection) {
            $result = $connection->reap_async_query();
    
            if ($result instanceof \mysqli_result) {
                // select query, do something with result
                $result->fetch_*...
    
                $result->free();
            } else {
               // write query like insert, update or ddl
            }
        }

        foreach ($error as $failed) {
            // do something with errors    
        }

        foreach ($reject as $_) {
            // do something with rejections    
        }
    } else {
        // Timeout - no results
    }
} while (\count($connections) !== \count($read) + \count($error) + \count($reject));

[kelunik]

Only with its own poll AFAIK, which is incompatible with a generic purpose event loop. We can only have one blocking operation waiting for events, not two.