arkouda
arkouda copied to clipboard
Closes #3054: Dynamically switch to batching for larger csv writes
This PR (closes #3054) adds the ability to write chunks of data when writing csv files that would cause us to run out of memory when using the new optimizatioin. We use the arkouda memory management functions to determine approximately how much memory is available on each locale, and then divide the data we want to write on that locale into slices of that size. This makes sure the chapel native strings array is as big as possible without running out of. Hopefully we'll keep the performance bump for the small cases