cli sf data query --bulk throws Fatal Error Javascript Heap out of memory

Is your feature request related to a problem? Please describe.

Bulk query stores all records in memory, instead of writing them to disk

src: https://github.com/jsforce/jsforce/blob/95932f288a59a0b68ff9fe8cc1a94523a4d0fb85/src/api/bulk.ts#L1077-L1099

src: https://github.com/jsforce/jsforce/blob/95932f288a59a0b68ff9fe8cc1a94523a4d0fb85/src/api/bulk.ts#L1244-L1264

this could throw an exception like Process out of Memory Exception if someone tries to export a few million records, because all records are stored in memory inside of a record[] array. For instance,

sf data query --query "SELECT Id FROM Contact" --bulk

src: https://github.com/salesforcecli/plugin-data/blob/b56e778f6c9ef8affbc0719cafb629b07d19ce2c/src/commands/data/query.ts#L135-L153

This type of exception can't be captured by a try/catch. The node runtime would exit 1 straight away.

This issue does not happen when using jsforce bulk v1 implementation because it was developed with streams in mind.

src: https://github.com/jsforce/jsforce/blob/95932f288a59a0b68ff9fe8cc1a94523a4d0fb85/src/api/bulk.ts#L959-L983

What are you trying to do Export a few million records using sfdx force:data:query --bulk

Describe the solution you'd like

1 . jsforce must accept a write stream so that memory could be released after its data is written to disk. Additionally, memory can be released when this.records[] reaches N kbs of data. 2. sfdx should create a write stream when the -b flag is true.

Describe alternatives you've considered N/a

Additional context N/a

Mar 12 '23 22:03 AllanOricil

Thank you for filing this feature request. We appreciate your feedback and will review the feature at our next grooming or sprint planning session. We prioritize feature requests with more upvotes and comments.