sbi
sbi copied to clipboard
Fix for joblib not saturating CPU during multiprocessing
What does this implement/fix? Explain your changes
This PR refactors thesimulate_for_sbi method so that now:
- it does not rely anymore on
simulate_in_batches - it provides a working solution for the multiprocessing issue explored in #1175 ...
Does this close any currently open issues?
Fixes #1175
Any relevant code examples, logs, error output, etc?
Refer to the thread in #1175 ...
Any other comments?
Further refactoring will be needed in order to avoid the process_simulator wrapping (possibly reducing the simulation runtime by roughly another 3 times)
...
Checklist
Put an x in the boxes that apply. You can also fill these out after creating
the PR. If you're unsure about any of them, don't hesitate to ask. We're here to
help! This is simply a reminder of what we are going to look for before merging
your code.
- [x] I have read and understood the contribution guidelines
- [x] I agree with re-licensing my contribution from AGPLv3 to Apache-2.0.
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have added tests that prove my fix is effective or that my feature works
- [x] I have reported how long the new tests run and potentially marked them
with
pytest.mark.slow. - [x] New and existing unit tests pass locally with my changes
- [x] I performed linting and formatting as described in the contribution guidelines
- [x] I rebased on
main(or there are no conflicts withmain) - [ ] For reviewer: The continuous deployment (CD) workflow are passing.