Could not process "/var/vhost/vendor/rector/rector/vendor/symplify/easy-parallel/src/ValueObject/ParallelProcess.php" file

[MetalArend]

Bug Report / Question?

Subject	Details
Rector version	0.13.8

I'm trying to run Rector against a fairly large and complex codebase, so it takes a lot of time to run a big set of rules. Either I use parallel, and it fails, or I disable parallel, and it takes waaaay too long.

Even with a fairly low parallel setup like $rectorConfig->parallel(120, 4, 1), it keeps giving the following errors:

[ERROR] Could not process "/var/vhost/vendor/rector/rector/vendor/symplify/easy-parallel/src/ValueObject/ParallelProcess.php" file, due to:                             
"Child process timed out after 120 seconds". On line: 103

[ERROR] Could not process some files, due to:                                  
"Reached system errors count limit of 50, exiting...".

Maybe this is more of a question than a bug report. The "Could not process" error is reporting on a Rector file from the vendor directory. But I have the vendor directory in the "skip" setup. Is it still always processing the vendor directory, without analysing it? Is there a way to tell Rector to completely ignore a certain directory? Even ignore it from the autoloading files? And would that help?

There's another possible bug in there as well, which is that whenever I run Rector and it fails with this error, it leaves something running in the background. But one bug at a time.

Minimal PHP Code Causing Issue

I have no way of giving you a reproducable code example for now, which annoys the hell out of me. I would be happy to help with adding any debug information, if there's a way to provide it. I have searched the issues, looked into the source code.

This is my config for now:

<?php

declare(strict_types=1);

use Rector\CodeQuality\Rector\ClassMethod\DateTimeToDateTimeInterfaceRector;
use Rector\CodeQuality\Rector\Foreach_\ForeachToInArrayRector;
use Rector\CodeQuality\Rector\Foreach_\SimplifyForeachToArrayFilterRector;
use Rector\CodeQuality\Rector\FuncCall\SimplifyRegexPatternRector;
use Rector\CodeQuality\Rector\If_\ExplicitBoolCompareRector;
use Rector\CodingStyle\Rector\Catch_\CatchExceptionNameMatchingTypeRector;
use Rector\CodingStyle\Rector\Class_\AddArrayDefaultToArrayPropertyRector;
use Rector\CodingStyle\Rector\ClassConst\VarConstantCommentRector;
use Rector\CodingStyle\Rector\ClassMethod\MakeInheritedMethodVisibilitySameAsParentRector;
use Rector\CodingStyle\Rector\ClassMethod\UnSpreadOperatorRector;
use Rector\CodingStyle\Rector\Encapsed\EncapsedStringsToSprintfRector;
use Rector\CodingStyle\Rector\FuncCall\ConsistentPregDelimiterRector;
use Rector\CodingStyle\Rector\Stmt\NewlineAfterStatementRector;
use Rector\Config\RectorConfig;
use Rector\Core\ValueObject\PhpVersion;
use Rector\Doctrine\Rector\Class_\RemoveRedundantDefaultClassAnnotationValuesRector;
use Rector\Doctrine\Set\DoctrineSetList;
use Rector\Php55\Rector\String_\StringClassNameToClassConstantRector;
use Rector\Php56\Rector\FuncCall\PowToExpRector;
use Rector\Php71\Rector\FuncCall\CountOnNullRector;
use Rector\Php73\Rector\FuncCall\JsonThrowOnErrorRector;
use Rector\Php74\Rector\Closure\ClosureToArrowFunctionRector;
use Rector\Php74\Rector\LNumber\AddLiteralSeparatorToNumberRector;
use Rector\Php80\Rector\Class_\AnnotationToAttributeRector;
use Rector\Php80\Rector\Class_\ClassPropertyAssignToConstructorPromotionRector;
use Rector\Privatization\Rector\Class_\FinalizeClassesWithoutChildrenRector;
use Rector\Set\ValueObject\LevelSetList;
use Rector\Set\ValueObject\SetList;
use Rector\Symfony\Set\SymfonyLevelSetList;
use Rector\Symfony\Set\SymfonySetList;
use Rector\Transform\Rector\MethodCall\ServiceGetterToConstructorInjectionRector;
use Rector\TypeDeclaration\Rector\ClassMethod\AddArrayParamDocTypeRector;
use Rector\TypeDeclaration\Rector\ClassMethod\AddArrayReturnDocTypeRector;
use Rector\TypeDeclaration\Rector\ClassMethod\AddParamTypeDeclarationRector;
use Rector\TypeDeclaration\Rector\ClassMethod\ArrayShapeFromConstantArrayReturnRector;
use Rector\TypeDeclaration\Rector\ClassMethod\ParamTypeByMethodCallTypeRector;
use Rector\TypeDeclaration\Rector\FunctionLike\ParamTypeDeclarationRector;
use Rector\TypeDeclaration\Rector\FunctionLike\ReturnTypeDeclarationRector;
use Rector\TypeDeclaration\Rector\Property\PropertyTypeDeclarationRector;

return static function (RectorConfig $rectorConfig): void {
//    $rectorConfig->disableParallel();
    $rectorConfig->parallel(10, 4, 1);
    $rectorConfig->paths([
        __DIR__ . '/src',
        __DIR__ . '/tests',
    ]);
    $rectorConfig->rules([
    ]);
    $rectorConfig->skip([
        __DIR__ . '/config/bundles.php',
        __DIR__ . '/data',
        __DIR__ . '/vendor',
        __DIR__ . '/vendor/rector/rector/vendor/symplify/easy-parallel/src/ValueObject/ParallelProcess.php',
        __DIR__ . '/var',
        // Up to PHP 8.0
        AddLiteralSeparatorToNumberRector::class,
        ClassPropertyAssignToConstructorPromotionRector::class,
        ClosureToArrowFunctionRector::class,
        CountOnNullRector::class,
        JsonThrowOnErrorRector::class,
        PowToExpRector::class,
        StringClassNameToClassConstantRector::class,
        // PHP
        DateTimeToDateTimeInterfaceRector::class,
        ForeachToInArrayRector::class => [
            __DIR__ . '/src/Core/Factories/CustomerFactory.php',
        ],
        SimplifyForeachToArrayFilterRector::class => [
            __DIR__ . '/src/Core/Entities/Quote/ParameterVersion.php',
        ],
        SimplifyRegexPatternRector::class,
        CatchExceptionNameMatchingTypeRector::class,
        VarConstantCommentRector::class,
        MakeInheritedMethodVisibilitySameAsParentRector::class,
        UnSpreadOperatorRector::class,
        AddArrayDefaultToArrayPropertyRector::class,
        AddArrayReturnDocTypeRector::class,
        AddArrayParamDocTypeRector::class,
        ReturnTypeDeclarationRector::class,
        PropertyTypeDeclarationRector::class,
        ParamTypeByMethodCallTypeRector::class,
        ParamTypeDeclarationRector::class,
        ExplicitBoolCompareRector::class,
        EncapsedStringsToSprintfRector::class => [
            __DIR__ . '/src/Core/LookupTables/LookupTable.php',
        ],
        ConsistentPregDelimiterRector::class,
        NewlineAfterStatementRector::class,
        FinalizeClassesWithoutChildrenRector::class,
        ArrayShapeFromConstantArrayReturnRector::class,
        // Up to Symfony 5.2
        AddParamTypeDeclarationRector::class => [
            __DIR__ . '/src/Services/CollectionTranslator.php',
        ],
        // Symfony
        AnnotationToAttributeRector::class,
        // Doctrine
        RemoveRedundantDefaultClassAnnotationValuesRector::class => [
            __DIR__ . '/src/Databases/SQL/Services/EventLogDatabaseService.php',
        ],
        ServiceGetterToConstructorInjectionRector::class => [
            __DIR__ . '/src/Databases/SQL/Services/EventLogDatabaseService.php',
        ],
    ]);
    $rectorConfig->phpVersion(PhpVersion::PHP_80);
    $rectorConfig->phpstanConfig(__DIR__ . '/phpstan.neon');
    $rectorConfig->importNames();
    //$rectorConfig->symfonyContainerPhp();

    $rectorConfig->sets([
//        // Up to PHP 8.0
//        LevelSetList::UP_TO_PHP_80,
//        // PHP
//        SetList::CODE_QUALITY,
//        SetList::CODING_STYLE,
//        // TODO SetList::DEAD_CODE,                                        Too many changes
//        // TODO SetList::EARLY_RETURN,                                     Too many changes
//        SetList::MYSQL_TO_MYSQLI,
//        // TODO SetList::PRIVATIZATION,                                    Too many changes
//        SetList::PSR_4,
//        SetList::TYPE_DECLARATION,
//        SetList::TYPE_DECLARATION_STRICT,
//        SetList::UNWRAP_COMPAT,
//        // Up to Symfony 5.2
        SymfonyLevelSetList::UP_TO_SYMFONY_52,
//        // Symfony
//        SymfonySetList::SYMFONY_52_VALIDATOR_ATTRIBUTES,
//        SymfonySetList::SYMFONY_CODE_QUALITY,
//        SymfonySetList::SYMFONY_CONSTRUCTOR_INJECTION,
//        SymfonySetList::SYMFONY_STRICT,
//        //SymfonySetList::ANNOTATIONS_TO_ATTRIBUTES,
//        // Doctrine
//        DoctrineSetList::DOCTRINE_BEHAVIORS_20,
//        DoctrineSetList::DOCTRINE_CODE_QUALITY,
//        DoctrineSetList::DOCTRINE_COMMON_20,
//        DoctrineSetList::DOCTRINE_DBAL_30,
//        DoctrineSetList::DOCTRINE_ORM_29,
//        // TODO DoctrineSetList::DOCTRINE_REPOSITORY_AS_SERVICE,           Can not understand our current setup
//        //DoctrineSetList::ANNOTATIONS_TO_ATTRIBUTES,
    ]);
};

Expected Behaviour

[TomasVotruba]

Hi, it looks like you're running Rector on /vendor. That's why it's so slow. Only run Rector on the code you own and that can change.

[MetalArend]

@TomasVotruba but my configuration is telling otherwise, only running it on the paths /src and /tests. Or am I missing something very obvious?

This bit:

    $rectorConfig->paths([
        __DIR__ . '/src',
        __DIR__ . '/tests',
    ]);

[TomasVotruba]

Interesting. Could you narrow the paths() to minimal file causing this?

[MetalArend]

Happily. But the error seems to hint to Could not process "/var/vhost/vendor/rector/rector/vendor/symplify/easy-parallel/src/ValueObject/ParallelProcess.php", which is a file from Rector itself. Might it be hiding another file, and is that error wrong?

[MetalArend]

Well, not exactly Rector itself, but a file in the project's vendor that is part of the Rector dependencies. Which should not be processed, as __DIR__ . '/vendor' is not in the paths portion, and is even in the skip portion.

[samsonasik]

For very large codebase, my suggestion is to split into multiple commands:

vendor/bin/rector process src/Controller
vendor/bin/rector process src/Model

etc

that should handle and help narrow which part that take long time to process.

[rafaelbernard]

I am experiencing the same with a not very large codebase (~1K files). I have both "Child process timeout" and "Could not process" messages.

I am trying those suggestions: https://github.com/rectorphp/rector/issues/7323#issuecomment-1198104137

[rafaelbernard]

So, interesting. Maybe two different problems in my case:

$rectorConfig->disableParallel();

Worked nice, and evidenced an issue with one of my files. After that, increasing the timeout was the fix to "Child process timeout". On my config:

$rectorConfig->parallel(240, 2);

So, now this is all good to me. It seems a problematic file along with a need to increase the timeout.

[TomasVotruba]

@rafaelbernard Increasing time on a big project makes sense :+1:

[MetalArend]

Still wondering why that Rector file is giving troubles, as it should not be a problem. It should not even be scanned. I was able to split the project up, but it feels strange that it would give me an error on how a Rector file itself is killing the process.

[MetalArend]

I saw the documentation. Whenever I am encountering this error next time, I'll open a new issue with some more debug information.

I have been working on the last project by splitting the project into separate directories in my makefile, and do parallel processing with make instead of Rector, and the laptop manages to do that just fine. But the parallel processing of Rector itself failed every time on the Rector file as reported above.

[TomasVotruba]

:+1:

If you can provide a reproducible repository on Github, we would be smarter about why this happens.. IMO this will be some cyclic reference between 2 files :)