Lean icon indicating copy to clipboard operation
Lean copied to clipboard

Make RandomDataGeneration parallel for performance gains

Open JosueNina opened this issue 6 months ago • 0 comments

Expected Behavior

The RandomDataGenerator should support parallel execution to improve performance. Ideally, tick generation for different symbols would run concurrently while maintaining deterministic output.

Actual Behavior

Currently, the generator runs in a single thread, processing each enumerator sequentially. For large datasets (e.g., multiple symbols over several days or months), this results in long execution times

For instance, running the following command: lean data generate --start 20231004 --end 20231020 --tickers=SPY --resolution=Minute --security-type=Option --verbose

Running this can take quite a long time

Potential Solution

Checklist

  • [x] I have completely filled out this template
  • [x] I have confirmed that this issue exists on the current master branch
  • [x] I have confirmed that this is not a duplicate issue by searching issues

JosueNina avatar Jun 25 '25 21:06 JosueNina