benkeen
All in memory - plans to write to disk?
If I understand it correctly, this is all generated in memory, even with 32Gb of RAM I can only generate c. 20m rows (changed all settings to use more memory, run longer etc). Is anybody working on enhancements that can make this write to disk once in a while which would allow it to write much larger data sets?
Hey @javiercr2020,
Great question and idea! The next version (pre-alpha, see generatedata4.com) moves everything into the javascript & away from the server, so the bottlenecks of having to go to and from the server will be eliminated. It'll run in web-workers in the browser, so the limits would now be your own PC. It should be far faster. It'll also be available to run as a node script, which again I think will be faster.
But yes, for really massive data sets writing to disk makes absolute sense. At this stage though I just need to get the new version out the door and then start performance testing it. But I'll absolutely keep this in mind.
Ben
