arkouda icon indicating copy to clipboard operation
arkouda copied to clipboard

Closes #3054: Dynamically switch to batching for larger csv writes

Open stress-tess opened this issue 11 months ago • 0 comments

This PR (closes #3054) adds the ability to write chunks of data when writing csv files that would cause us to run out of memory when using the new optimizatioin. We use the arkouda memory management functions to determine approximately how much memory is available on each locale, and then divide the data we want to write on that locale into slices of that size. This makes sure the chapel native strings array is as big as possible without running out of. Hopefully we'll keep the performance bump for the small cases

stress-tess avatar Mar 22 '24 21:03 stress-tess