sbi icon indicating copy to clipboard operation
sbi copied to clipboard

Fix for joblib not saturating CPU during multiprocessing

Open janko-petkovic opened this issue 1 year ago • 0 comments

What does this implement/fix? Explain your changes

This PR refactors thesimulate_for_sbi method so that now:

  • it does not rely anymore on simulate_in_batches
  • it provides a working solution for the multiprocessing issue explored in #1175 ...

Does this close any currently open issues?

Fixes #1175

Any relevant code examples, logs, error output, etc?

Refer to the thread in #1175 ...

Any other comments?

Further refactoring will be needed in order to avoid the process_simulator wrapping (possibly reducing the simulation runtime by roughly another 3 times) ...

Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.

  • [x] I have read and understood the contribution guidelines
  • [x] I agree with re-licensing my contribution from AGPLv3 to Apache-2.0.
  • [x] I have commented my code, particularly in hard-to-understand areas
  • [x] I have added tests that prove my fix is effective or that my feature works
  • [x] I have reported how long the new tests run and potentially marked them with pytest.mark.slow.
  • [x] New and existing unit tests pass locally with my changes
  • [x] I performed linting and formatting as described in the contribution guidelines
  • [x] I rebased on main (or there are no conflicts with main)
  • [ ] For reviewer: The continuous deployment (CD) workflow are passing.

janko-petkovic avatar Jul 01 '24 16:07 janko-petkovic