Drall stuck when running a Drush command on a remote site inside a Docker container

[beerendlauwers]

I'm using Drall v2.1.0.

Running the command vendor/bin/drall exec drush @@site.T cr inside a container derived from php:8.1.18-cli spawns workers, but they don't return the result or exit code:

https://github.com/jigarius/drall/assets/3713919/caa69433-0a34-4934-91b3-4626684ce6cc

My version has the code committed for https://github.com/jigarius/drall/issues/71, so I'm not sure what the issue could be.

[jigarius]

Hey! This seems to be the same as https://github.com/jigarius/drall/issues/58. I will try and take a look at this asap. Thanks.

In the meantime, can you try that command with a -y flag and see if that works?

[beerendlauwers]

No difference, unfortunately. I've also tried rearranging the Drall flags, I saw that putting them after the command isn't officially supported.

https://github.com/jigarius/drall/assets/3713919/828b5578-0a21-4163-8e20-1c87b76207ee

[beerendlauwers]

I see the workers have been written with ConcurrentIterator\each. Could it also use parallel Workers from https://github.com/amphp/parallel ?

[beerendlauwers]

Using that library didn't change things. However, I found an interesting example to pipe output: https://github.com/amphp/parallel/blob/1.x/examples/process.php

Replacing $sOutput = yield ByteStream\buffer($process->getStdout()); in BaseExecCommand with Promise\rethrow(ByteStream\pipe($process->getStdout(), ByteStream\getStdout())); (also add use Amp\Promise;) gets me output:

https://github.com/jigarius/drall/assets/3713919/60f926aa-9649-4de8-b534-f4f248932cc2

Then entire process still hangs, unfortunately.

[beerendlauwers]

I suspected that the issue was within Drush, as I've encountered a similar issue with remote commands not finishing in another situation.

We're encountering the same problem as this person: https://stackoverflow.com/questions/60694192/unittesting-a-symfony-4-2-process-runs-infinite-loop-than-times-out-wihout-un

Namely, in Symfony\Component\Process\Process::wait(), there's this code:

  $this->requireProcessIsStarted(__FUNCTION__);

  $this->updateStatus(false);

  if (null !== $callback) {
      if (!$this->processPipes->haveReadSupport()) {
          $this->stop(0);
          throw new LogicException('Pass the callback to the "Process::start" method or call enableOutput to use a callback with "Process::wait".');
      }
      $this->callback = $this->buildCallback($callback);
  }

  do {
      $this->checkTimeout();
      $running = '\\' === \DIRECTORY_SEPARATOR ? $this->isRunning() : $this->processPipes->areOpen();
      $this->readPipes($running, '\\' !== \DIRECTORY_SEPARATOR || !$running);
  } while ($running);

  while ($this->isRunning()) {
      $this->checkTimeout();
      usleep(1000);
  }

  if ($this->processInformation['signaled'] && $this->processInformation['termsig'] !== $this->latestSignal) {
      throw new ProcessSignaledException($this);
  }

  return $this->exitcode;

The part that is tripping us up is this:

do {
    $this->checkTimeout();
    $running = '\\' === \DIRECTORY_SEPARATOR ? $this->isRunning() : $this->processPipes->areOpen();
    $this->readPipes($running, '\\' !== \DIRECTORY_SEPARATOR || !$running);
} while ($running);
}

Printing a . in this loop produces this behaviour:

https://github.com/jigarius/drall/assets/3713919/ebe17da8-2a0a-4dcd-9ffb-34003ec21292

After the remote command has completed, $this->processPipes->areOpen() remains true and it loops until Drush's default timeout is hit (4 hours).

The solution is simple: use $this->isRunning() instead of checking the pipes. isRunning() calls updateStatus(false), which closes the process if it's finished.

So, replacing

$running = '\\' === \DIRECTORY_SEPARATOR ? $this->isRunning() : $this->processPipes->areOpen();

with

$running = $this->isRunning();

Gets us this:

https://github.com/jigarius/drall/assets/3713919/33b403f6-c102-4189-acad-60dc534a6fe7

[beerendlauwers]

Seems like that line was introduced here: https://github.com/symfony/process/commit/2ab81276d842174a51f4eb002332f41b90c876cf?diff=split

I'm not entirely sure why this was refactored this way. Before, $this->processPipes->hasOpenHandles() was used to handle WIndows-specific code. After the refactor, it's solely used by non-Windows code...

I'll see if I can open an issue on their repository.

EDIT: There was already one present, I've mentioned my research.

[jigarius]

Thanks a tonne for the research. I'll look into this as soon as I get time. I'll have to find a project on which I can use remote aliases first.

[beerendlauwers]

@jigarius It might be possible to get this fixed in Drush instead of Symfony. The maintainer seems to be fine with it if we can prove it doesn't break anything.

[jigarius]

Sounds good to me. BTW, I just pushed the first 3.x tag. If you have PHP 8.1, then I invite you to try it. There are many cool new features.

As for this issue, I'll do some research this week. For now, maybe you can SSH into the remote container and then run Drall locally.

[lems3]

On my side, even by getting in the container sometime it will get stuck.

The only way we found to skip that is to skip interactive questions with --no-interactive.

We use Lando for our local environment, and we don't have to SSH into our containers as long as we're patient and we use the --no-interactive flag. (patient not because of speed but because feedbacks is shown only once after each site is done).

[jigarius]

@lems3 I see. Since Drall uses a 3rd-party library for running commands in parallel, it is subject to the limitations of that library.

The output buffering was added to prevent the output of multiple Drush commands (running in parallel) from mixing with each other. There is an issue for disabling the output buffer; see #57.

As for the --no-interactive, you should be able to use -y instead. I would suggest running lando ssh before running Drall. This is because commands like lando drall drush ... can sometimes have side-effects on the command being passed into the container.