autoemulate
autoemulate copied to clipboard
Set random seeds for Emulator models
Closes #465
TODO
- [x] Branches from #464 so merge that first
- ~~Ensure all emulators using the
set_random_seedfunc~~ some models (LGBM and RandomForest) pass therandom_seedint into the model params, so presumably you don't need to do both - [ ] Ensure all emulators have tests for checking the predictions are consistent when the
random_seedis set - [x] Remove the seed being set separately in
CNPDataset- Still doesn't result in
test_fit_predict_deterministic_with_seedpassing
- Still doesn't result in
- [ ] Consider refactoring so the seed is set at an even higher level (see comment below)
- [ ] Why does
test_fit_predict_deterministic_with_seedalways pass regardless ofrandom_statefor gaussian exact? - [ ] May want to add a new issue to tackle later to ensure that tests pass for determinism on GPUs
Reviewers
@EdwinB12 if you have a better sense than me of how CNP/Gaussian processes work and how random seeds should affect the result would be great to get your review here. See my comments below, but my key questions are:
- Why does
test_fit_predict_deterministic_with_seedfail for CNP?- There must be some randomness being introduced here still, but I can't see where
- Why does
test_fit_predict_deterministic_with_seedalways pass regardless ofrandom_statefor GP?
Coverage report
This report was generated by python-coverage-comment-action
Click to see where and how coverage changed
File Statements Missing Coverage Coverage
(new stmts)Lines missing
autoemulate/experimental
model_selection.py
autoemulate/experimental/emulators
base.py
lightgbm.py
random_forest.py
svm.py
autoemulate/experimental/emulators/gaussian_process
exact.py
autoemulate/experimental/emulators/neural_processes
conditional_neural_process.py
tests
test_compare.py
tests/experimental
test_experimental_base.py
test_experimental_gaussian_process_exact.py
test_experimental_lightgbm.py
test_experimental_random_forest.py
test_experimental_svm.py
Project Total
Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:
Project coverage is 81.48%. Comparing base (
71aba97) to head (fcc9357).
Additional details and impacted files
@@ Coverage Diff @@
## main #479 +/- ##
==========================================
+ Coverage 81.28% 81.48% +0.20%
==========================================
Files 107 107
Lines 7420 7503 +83
==========================================
+ Hits 6031 6114 +83
Misses 1389 1389
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
:rocket: New features to boost your workflow:
- :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.