Source tracking fails on larger projects

[nkadis]

Summary

There appears to be an issue with source tracking when deploying a large project (tested with ~13k components). Deployments succeeded after the fix to source deploy for ~9k components, but at ~13k it fails to update source tracking.

Steps To Reproduce:

Repository to reproduce: Can't reproduce in dreamhouse-lwc, but any large project should do (mine pushes 13074 components)

1. Run sfdx force:source:push -w 90 --json to deploy a large component set (wait time added due to amount of components being deployed).

Expected result

Deployment completes successfully and source tracking updated.

Actual result

Deployment completes successfully, but source tracking fails.

System Information

• Which shell/terminal are you using? (e.g. bash, zsh, powershell 5, powershell 7, cmd.exe, etc.) zsh

• sfdx version --verbose --json:

{
  "cliVersion": "sfdx-cli/7.166.1",
  "architecture": "darwin-arm64",
  "nodeVersion": "node-v16.15.0",
  "pluginVersions": [
    "@oclif/plugin-autocomplete 1.3.0 (core)",
    "@oclif/plugin-commands 2.2.0 (core)",
    "@oclif/plugin-help 5.1.12 (user)",
    "@oclif/plugin-not-found 2.3.1 (core)",
    "@oclif/plugin-plugins 2.1.0 (core)",
    "@oclif/plugin-update 3.0.0 (core)",
    "@oclif/plugin-version 1.1.2 (core)",
    "@oclif/plugin-warn-if-update-available 2.0.4 (user)",
    "@oclif/plugin-which 2.1.0 (core)",
    "alias 2.1.0 (core)",
    "apex 1.2.0 (core)",
    "auth 2.2.3 (core)",
    "community 2.0.1 (core)",
    "config 1.4.17 (core)",
    "custom-metadata 2.0.0 (core)",
    "data 2.1.2 (core)",
    "generator 2.0.2 (core)",
    "info 2.0.1 (core)",
    "limits 2.0.1 (core)",
    "org 2.2.0 (core)",
    "packaging 1.5.1 (core)",
    "schema 2.1.1 (core)",
    "signups 1.2.0 (core)",
    "source 2.0.13 (core)",
    "telemetry 2.0.0 (core)",
    "templates 55.1.0 (core)",
    "trust 2.0.3 (core)",
    "user 2.1.0 (core)",
    "@salesforce/sfdx-plugin-lwc-test 1.0.0 (core)",
    "@salesforce/sfdx-scanner 2.13.8 (user)",
    "salesforce-alm 54.8.1 (core)"
  ],
  "osVersion": "Darwin 21.6.0",
  "shell": "zsh",
  "rootPath": "/Users/nicolaskadis/.nvm/versions/node/v16.15.0/lib/node_modules/sfdx-cli"
}

Additional information

Deployment status succeeded:

Output from command:

{
  "result": [
    {
      "code": "NotFoundError",
      "data": {
        "what": "force-app/main/default/objects/Custom_Object__c/fields/Custom_Field__c.field-meta.xml"
      },
      "caller": "",
      "message": "Could not find force-app/main/default/objects/Custom_Object__c/fields/Custom_Field__c.field-meta.xml.",
      "stack": "NotFoundError: Could not find force-app/main/default/objects/Custom_Object__c/fields/Custom_Field__c.field-meta.xml.
        at /Users/nicolaskadis/.nvm/versions/node/v16.15.0/lib/node_modules/sfdx-cli/node_modules/isomorphic-git/index.cjs:4518:34"
    },
    ...
  ],
  "stack": "MultipleGit: There are multiple errors that were thrown by the method. Please refer to the \"errors\" property to see more\n
    at ShadowRepo.commitChanges (/Users/nicolaskadis/.nvm/versions/node/v16.15.0/lib/node_modules/sfdx-cli/node_modules/@salesforce/source-tracking/lib/shared/localShadowRepo.js:194:39)
    at async SourceTracking.updateLocalTracking (/Users/nicolaskadis/.nvm/versions/node/v16.15.0/lib/node_modules/sfdx-cli/node_modules/@salesforce/source-tracking/lib/sourceTracking.js:267:9)
    at async Promise.all (index 0)
    at async updateTracking (/Users/nicolaskadis/.nvm/versions/node/v16.15.0/lib/node_modules/sfdx-cli/node_modules/@salesforce/plugin-source/lib/trackingFunctions.js:74:5)
    at async Push.updateTracking (/Users/nicolaskadis/.nvm/versions/node/v16.15.0/lib/node_modules/sfdx-cli/node_modules/@salesforce/plugin-source/lib/commands/force/source/push.js:102:9)
    at async Push.run (/Users/nicolaskadis/.nvm/versions/node/v16.15.0/lib/node_modules/sfdx-cli/node_modules/@salesforce/plugin-source/lib/commands/force/source/push.js:31:9)
    at async Push._run (/Users/nicolaskadis/.nvm/versions/node/v16.15.0/lib/node_modules/sfdx-cli/node_modules/@salesforce/command/lib/sfdxCommand.js:102:40)
    at async Config.runCommand (/Users/nicolaskadis/.nvm/versions/node/v16.15.0/lib/node_modules/sfdx-cli/node_modules/@oclif/core/lib/config/config.js:272:25)
    at async run (/Users/nicolaskadis/.nvm/versions/node/v16.15.0/lib/node_modules/sfdx-cli/node_modules/@oclif/core/lib/main.js:76:5)",
  "warnings": [],
  "commandName": "Push"
}

Note: truncated result list since all errors are the same (all ~9580 of them) and prettified stacks.

[github-actions[bot]]

Thank you for filing this issue. We appreciate your feedback and will review the issue as soon as possible. Remember, however, that GitHub isn't a mechanism for receiving support under any agreement or SLA. If you require immediate assistance, contact Salesforce Customer Support.

[ghost]

I also received this error. However, I also get it when running sfdx force:source:tracking:reset. The cli was recently upgraded from 7.143 to 7.161. After doing this, I went to follow the instructions of migrating to use the newer commands, by running legacy:push, and then clearing legacy tracking. After this, I attempt to run tracking:reset and get the error as described above.

[nkadis]

Can confirm that the error also occurs with sfdx force:source:tracking:reset as @nBray-tLP described.

[booknownick]

Glad this is not just me, can confirm I also receive this error. Package contains a total of 5000 components.

These pushes are to a scratch org, if that helps at all. Have attempted to move scratch org, re-install VS code, cli, git ETC.

"code": "NotFoundError", "data": { "what": "force-app/main/default/workflows/Booking__c.workflow-meta.xml" }, "caller": "", "message": "Could not find force-app/main/default/workflows/Booking__c.workflow-meta.xml.", "stack": "NotFoundError: Could not find force-app/main/default/workflows/Booking__c.workflow-meta.xml.\n at /usr/local/lib/sfdx/node_modules/isomorphic-git/index.cjs:4518:34" }, { "code": "NotFoundError", "data": { "what": "force-app/main/default/workflows/List_Outcome__c.workflow-meta.xml" }, "caller": "", "message": "Could not find force-app/main/default/workflows/List_Outcome__c.workflow-meta.xml.", "stack": "NotFoundError: Could not find force-app/main/default/workflows/List_Outcome__c.workflow-meta.xml.\n at /usr/local/lib/sfdx/node_modules/isomorphic-git/index.cjs:4518:34" } ], "stack": "MultipleGit: There are multiple errors that were thrown by the method. Please refer to the "errors" property to see more\n at ShadowRepo.commitChanges (/usr/local/lib/sfdx/node_modules/@salesforce/source-tracking/lib/shared/localShadowRepo.js:194:39)\n at async SourceTracking.updateLocalTracking (/usr/local/lib/sfdx/node_modules/@salesforce/source-tracking/lib/sourceTracking.js:267:9)\n at async Promise.all (index 0)\n at async updateTracking (/usr/local/lib/sfdx/node_modules/@salesforce/plugin-source/lib/trackingFunctions.js:74:5)\n at async Push.updateTracking (/usr/local/lib/sfdx/node_modules/@salesforce/plugin-source/lib/commands/force/source/push.js:102:9)\n at async Push.run (/usr/local/lib/sfdx/node_modules/@salesforce/plugin-source/lib/commands/force/source/push.js:31:9)\n at async Push._run (/usr/local/lib/sfdx/node_modules/@salesforce/command/lib/sfdxCommand.js:102:40)\n at async Config.runCommand (/usr/local/lib/sfdx/node_modules/@oclif/core/lib/config/config.js:272:25)\n at async run (/usr/local/lib/sfdx/node_modules/@oclif/core/lib/main.js:76:5)", "warnings": [], "commandName": "Push" }

[WillieRuemmele]

Hi @nkadis @booknownick @nBray-tLP sorry for the delayed response.

I'm assuming this issue is still happening. As a workaround you can try the force:source:deploy command with --tracksource which should be equivalent to a source:push. Additionally, if that doesn't work, you can run the source:deploy command, and then run force:source:tracking:reset command, to get you in a clean source-tracking state.

Please let me know if either of those work

[booknownick]

Hey @WillieRuemmele Can confirm issue is still present, although all of our devs have switched over to using the deploy command for the meantime, as cumbersome as it is! Just to confirm, --tracksource will only push changes made if we use force:source:deploy with the path as 'force-app/main/default' ?

Many thanks!

[WillieRuemmele]

--tracksource should update the source tracking for whatever was in that deployment regardless of how you specify it

[booknownick]

Okay, so we still have to push individual files in the interim, until we have a fix for this issue

[nkadis]

After some fiddling around, I believe we found the issue being caused by forcedotcom/source-tracking/src/shared/localShadowRepo.ts:62.

this.maxFileAdd = this.isWindows ? 8000 : 15000;

This is used to form the chunks/batches to be tracked (using git.add). On Windows, the max chunk size is set to 8000, but for anything else, its set to 15000. If you change the value to be 8000 for all other platforms too, then the error stops getting thrown.

We've tested this on Mac, so I can't say if this issue also occurs on Linux etc. and we haven't tested if this would work with any other value between 8000 and 15000, but at least we have a temporary fix.

[cromwellryan]

Hello friends! And thank you for your help!

Can you please post the results of the following commands on the offending system:

ulimit -n ulimit -Sn ulimit -Hn

For context, we believe our batch size (maxFileAdd) during source track updates are higher than hard open file limits on some systems.

[rmathewsbeyond]

we are getting the same source-tracking errors, we are on sfdx-cli/7.182.1 linux-x64 node-v16.13.1 and the max file open limit currently is 4096

[iowillhoit]

Thanks @rmathewsbeyond

@cromwellryan was finally able to replicate this issue! Big thanks to @nkadis for determining where this issue was originating!

It appears to be because the number of files that source tracking is batching is exceeding your user specific file limits.

Try one of the following:

• Increase your hard file limit in the same session with ulimit -Hn 20000
• Or install the prerelease of plugin-source with sfdx plugins:install @salesforce/plugin-source@dev and then set the SFDX_SOURCE_TRACKING_BATCH_SIZE environment variable to a value lower than the output of ulimit -Hn
  • A few notes:
    • Don’t set the env var too close to the output of ulimit -Hn or your system will still hit that limit. If you continue to get the error, lower the value of the env var some more.
    • We will be releasing these changes in next week's RC

[rmathewsbeyond]

this fix worked, thanks.

Thanks @rmathewsbeyond

@cromwellryan was finally able to replicate this issue! Big thanks to @nkadis for determining where this issue was originating!

It appears to be because the number of files that source tracking is batching is exceeding your user specific file limits.

Try one of the following:

• Increase your hard file limit in the same session with ulimit -Hn 20000

• Or install the prerelease of plugin-source with sfdx plugins:install @salesforce/plugin-source@dev and then set the SFDX_SOURCE_TRACKING_BATCH_SIZE environment variable to a value lower than the output of ulimit -Hn

  • A few notes:

    • Don’t set the env var too close to the output of ulimit -Hn or your system will still hit that limit. If you continue to get the error, lower the value of the env var some more.
    • We will be releasing these changes in next week's RC

[jeenamsuneja]

Getting this error Pipelinel echo 20:01:30 20:01:30 20:01:30 20:01:30 20:01:30 20:01:30 20:01:30 20:01:30 20:01:30 20:01:30 20:01:30 20:01:30 "status": 1, "result": { "done": true, "errorMessage": "LIMIT_EXCEEDED: Too many files in retrieve call, limit is: 10000", "errorStatusCode": "LIMIT_EXCEEDED" , "status": "Failed", "success": false, "zipFile": { "$"：｛ "xi:nil": "true" 20:01:30 」 20:01:30 ｝ 20:01:30 20:01:30 20:01:30 20:01:30 20:01:30 "MdapiRetrieveCommand", 20:01:30 20:01:30 20:01:30 20:01:30 20:01:30 20:01:30 20:01:30 20:01:30 20:01:30 20:01:30 "name" "mdapiRetrieveFailed", "message": "The metadata retrieve operation failed: LIMIT_EXCEEDED: Too many files in retrieve ca. "exitCode": 1, "commandName" : "data": { "done": true, "errorMessage": "LIMIT_EXCEEDED: Too many files in retrieve call, limit is: 10000", "errorStatusCode": "LIMIT_EXCEEDED", "status": "Failed", "success": false, "zipFile": { "$": { "xsi:nil": "true"