apex get test --codecoverage runs 30x times slower or crashes with JavaScript heap out of memory

[maksym-petrov-ct]

Summary

Compared to sfdx 7.x versions, sf versions (tested on 2.52 and 2.68.6) can't effectively run sf apex get test --codecoverage on org with hundreds of class files. It either crashes after ~30 min, or if successful, takes ~30 min. sfdx 7.x with force:apex:test:report --codecoverage takes 1-2 min and never crashes here.

Expected result

1. generate test results with code coverage (works as expected):

sf apex run test --wait 1 --result-format json --code-coverage --json -o TargetOrg --test-level RunSpecifiedTests --tests "Test_1,Test_2..."

2. after tests have fully finished, I'm getting test-run-id and running below command:

sf apex get test -o TargetOrg --code-coverage --test-run-id 70700000000000000 --json  --outputdir ./test_results > stdout.txt

in result I expect in 1-2 min to get test_results directory containing files:

test-result.txt
test-result-70700000000000000.json
test-result-70700000000000000-codecoverage.json
test-result-70700000000000000-junit.xml
test-result-codecoverage.json
test-run-id.txt

note that I compared behavior with sfdx v7.180 and sfdx gets job done with it's force:apex:test:report in 1-2 min.

Actual result

after having code coverage stats in org I can get it's test-run-id, something like 7070000000000000X. Some tests have failed in this test-run and overall coverage is below 75%, if it matters (but I think it doesn't).

Note that sf command is passed through 'time' and below is stats for system resources usage during execution (29 min of real time while only 1m23s of active time).

time sf apex get test -o TargetOrg --code-coverage --test-run-id 70700000000000000 --json  --outputdir ./test_results > stdout.txt

<--- Last few GCs --->

[508145:0x35ea3800]  1734278 ms: Scavenge 2033.6 (2071.1) -> 2030.4 (2072.1) MB, 3.74 / 0.00 ms  (average mu = 0.192, current mu = 0.131) allocation failure; 
[508145:0x35ea3800]  1734286 ms: Scavenge 2034.6 (2072.1) -> 2031.5 (2073.4) MB, 4.07 / 0.00 ms  (average mu = 0.192, current mu = 0.131) allocation failure; 
[508145:0x35ea3800]  1734292 ms: Scavenge 2036.1 (2073.6) -> 2033.0 (2074.9) MB, 3.61 / 0.00 ms  (average mu = 0.192, current mu = 0.131) allocation failure; 


<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
----- Native stack trace -----

 1: 0xb82c28 node::OOMErrorHandler(char const*, v8::OOMDetails const&) [node]
 2: 0xeed540 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node]
 3: 0xeed827 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node]
 4: 0x10ff3c5  [node]
 5: 0x10ff954 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [node]
 6: 0x1116844 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*) [node]
 7: 0x111705c v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 8: 0x11191ba v8::internal::Heap::HandleGCRequest() [node]
 9: 0x1084827 v8::internal::StackGuard::HandleInterrupts() [node]
10: 0x1526f7a v8::internal::Runtime_StackGuard(int, unsigned long*, v8::internal::Isolate*) [node]
11: 0x7fca2f699ef6 
Aborted (core dumped)

real    29m2.633s
user    1m23.516s
sys     0m9.384s

Additional information

after ~30 min sometimes test_results directory contains expected files, but sometimes not. Reproduced on MacOS, Ubuntu. Tested on version @salesforce/cli/2.52 - issue reproduces.

System Information

{
  "architecture": "wsl-x64",
  "cliVersion": "@salesforce/cli/2.68.6",
  "nodeVersion": "node-v20.13.1",
  "osVersion": "Linux 5.15.167.4-microsoft-standard-WSL2",
  "rootPath": "/home/ubuntu/.nvm/versions/node/v20.13.1/lib/node_modules/@salesforce/cli",
  "shell": "bash",
  "pluginVersions": [
    "@oclif/plugin-autocomplete 3.2.10 (core)",
    "@oclif/plugin-commands 4.1.10 (core)",
    "@oclif/plugin-help 6.2.18 (core)",
    "@oclif/plugin-not-found 3.2.28 (core)",
    "@oclif/plugin-plugins 5.4.17 (core)",
    "@oclif/plugin-search 1.2.16 (core)",
    "@oclif/plugin-update 4.6.13 (core)",
    "@oclif/plugin-version 2.2.16 (core)",
    "@oclif/plugin-warn-if-update-available 3.1.23 (core)",
    "@oclif/plugin-which 3.2.19 (core)",
    "@salesforce/cli 2.68.6 (core)",
    "@salesforce/commerce 252.0.0 (user) published 181 days ago (Thu Jun 13 2024)",
    "apex 3.6.3 (core)",
    "api 1.3.2 (core)",
    "auth 3.6.75 (core)",
    "community 3.2.32 (user) published 88 days ago (Sat Sep 14 2024) (latest is 3.3.6)",
    "data 3.11.4 (core)",
    "deploy-retrieve 3.15.13 (core)",
    "info 3.4.21 (core)",
    "limits 3.3.40 (core)",
    "marketplace 1.3.6 (core)",
    "org 5.2.4 (core)",
    "packaging 2.9.3 (core)",
    "schema 3.3.42 (core)",
    "settings 2.4.6 (core)",
    "sobject 1.4.46 (core)",
    "telemetry 3.6.23 (core)",
    "templates 56.3.30 (core)",
    "trust 3.7.43 (core)",
    "user 3.6.3 (core)",
    "@salesforce/sfdx-scanner 4.2.0 (user) published 197 days ago (Tue May 28 2024) (latest is 4.7.0)",
    "sfdx-git-delta 5.49.3 (user) published 29 days ago (Tue Nov 12 2024) (latest is 5.50.0)"
  ]
}