Intermittent error during scratch org pool generation

[sumanth-bolledla-acn]

Describe the bug Intermittently during Scratch org pool generation, the job completes successfully but a couple of scratch orgs fail to be created. There are no logs generated in .sfpowerscripts folder to check what was going wrong for the scratch org that failed. The only error indicator that we see was in the UI during the job execution.

To Reproduce Steps to reproduce the behavior:

1. In GitLab Cloud, set up project-template with dxatscale-template for GitLabs
2. Run the Scratch-Org pool generation job- (Assuming there are 3 SO per batch and there are 7 batches per job)
3. Intermittently for few batches, the scratch org creation fails for a scratch-org in the batch while the remaining two are created successfully

Expected behavior All scratch-orgs in to be created without any issue. If an error occurs, the error shows that it failed to create SO, but doesn't show why it failed.

Screenshots

In this image, the scratch#2 failed to create, while the rest were successful

Platform Details (please complete the following information):

• OS: GitLab.com Shared Runners
• Version [e.g. CLI Version eg: 1.6.6]
• Salesforce CLI(sfdx cli) Version: 129
• CI Platform: GitLab.com

Additional context There is no more info logged regarding the error

[azlam-abdulsalam]

@sumanth-bolledla-acn this is possibly a salesforce error?. Is this still frequent

[jonathankretzmer]

Could this possibly be due to a wait timeout when creating the scratch org? This seemed to happen far less frequently when I ran a linked version of sfpowerscripts with a 20-minute wait (from the hard-coded 6 minutes).

[azlam-abdulsalam]

Yes, this happens when scratch org creation times out, or fail to create. We need to address this by

• Wait time as a parameter in pool config, probably rhis means current model of sequentially creating all scratch orga before preparing it in parallel has to change(as some scratch org will be stuck, and resulting in long time to prepare)
• Fix error handling

[jonathankretzmer]

I looked at the scope of a comprehensive PR that implemented an optional parameter but the depth of changes really became extremely cumbersome. With your point about changes being made to change the behaviour to create all the Scratch Orgs in parallel (am excited about this!) it feels like the I'm glad I made that choice as it would have been refactored away anyway.

As a temporary fix, as I am seeing this as a systemic behaviour, I have submitted a PR to extend the duration to 20 minutes. As noted above, I did consider a more comprehensive change, but this was abandoned for now. See: Increase wait time for Scratch Org creation to 20 minutes #863

Logs showing this to be a systemic bug on our side:

Generate Scratch Orgs..
Creating Scratch  Org  1 of 3..
EE SyntaxError: Unexpected end of JSON input
    at JSON.parse (<anonymous>)
    at CreateScratchOrgImpl.<anonymous> (/sfdx_plugins/.local/share/sfdx/node_modules/@dxatscale/sfpowerscripts.core/lib/command/SFDXCommand.js:55:29)
    at Generator.next (<anonymous>)
    at fulfilled (/sfdx_plugins/.local/share/sfdx/node_modules/@dxatscale/sfpowerscripts.core/lib/command/SFDXCommand.js:24:58)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
SyntaxError: Unexpected end of JSON input
Unable to provision scratch org  1 ..   
Creating Scratch  Org  2 of 3..
EE SyntaxError: Unexpected end of JSON input
    at JSON.parse (<anonymous>)
    at CreateScratchOrgImpl.<anonymous> (/sfdx_plugins/.local/share/sfdx/node_modules/@dxatscale/sfpowerscripts.core/lib/command/SFDXCommand.js:55:29)
    at Generator.next (<anonymous>)
    at fulfilled (/sfdx_plugins/.local/share/sfdx/node_modules/@dxatscale/sfpowerscripts.core/lib/command/SFDXCommand.js:24:58)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
SyntaxError: Unexpected end of JSON input
Unable to provision scratch org  2 ..   
Creating Scratch  Org  3 of 3..
EE SyntaxError: Unexpected end of JSON input
    at JSON.parse (<anonymous>)
    at CreateScratchOrgImpl.<anonymous> (/sfdx_plugins/.local/share/sfdx/node_modules/@dxatscale/sfpowerscripts.core/lib/command/SFDXCommand.js:55:29)
    at Generator.next (<anonymous>)
    at fulfilled (/sfdx_plugins/.local/share/sfdx/node_modules/@dxatscale/sfpowerscripts.core/lib/command/SFDXCommand.js:24:58)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
SyntaxError: Unexpected end of JSON input
Unable to provision scratch org  3 ..   

[Schuchie]

Hey guys, can confirm that. I also receiving the mentioned error. Is there a workaround from the configuration site? Typically we set our wait time to 30 minutes.

I think the following file should be adjusted: https://github.com/Accenture/sfpowerscripts/blob/7842d2d1c73628ebe64f518e5cf5d39a51326582/packages/core/src/sfdxwrappers/CreateScratchOrgImpl.ts#L31

[azlam-abdulsalam]

I have partially address this in #903, this will now provide you with an option in the pool config to set the wait time. We will eventually move this to individual threads

[azlam-abdulsalam]

Will close this, a much more targeted issue is now available