autoemulate icon indicating copy to clipboard operation
autoemulate copied to clipboard

Set random seeds for Emulator models

Open edwardchalstrey1 opened this issue 6 months ago • 2 comments

Closes #465

TODO

  • [x] Branches from #464 so merge that first
  • ~~Ensure all emulators using the set_random_seed func~~ some models (LGBM and RandomForest) pass the random_seed int into the model params, so presumably you don't need to do both
  • [ ] Ensure all emulators have tests for checking the predictions are consistent when the random_seed is set
  • [x] Remove the seed being set separately in CNPDataset
    • Still doesn't result in test_fit_predict_deterministic_with_seed passing
  • [ ] Consider refactoring so the seed is set at an even higher level (see comment below)
  • [ ] Why does test_fit_predict_deterministic_with_seed always pass regardless of random_state for gaussian exact?
  • [ ] May want to add a new issue to tackle later to ensure that tests pass for determinism on GPUs

Reviewers

@EdwinB12 if you have a better sense than me of how CNP/Gaussian processes work and how random seeds should affect the result would be great to get your review here. See my comments below, but my key questions are:

  • Why does test_fit_predict_deterministic_with_seed fail for CNP?
    • There must be some randomness being introduced here still, but I can't see where
  • Why does test_fit_predict_deterministic_with_seed always pass regardless of random_state for GP?

edwardchalstrey1 avatar May 21 '25 10:05 edwardchalstrey1

Coverage report

Click to see where and how coverage changed
FileStatementsMissingCoverageCoverage
(new stmts)
Lines missing
  autoemulate/experimental
  model_selection.py
  autoemulate/experimental/emulators
  base.py
  lightgbm.py
  random_forest.py
  svm.py
  autoemulate/experimental/emulators/gaussian_process
  exact.py
  autoemulate/experimental/emulators/neural_processes
  conditional_neural_process.py
  tests
  test_compare.py
  tests/experimental
  test_experimental_base.py
  test_experimental_gaussian_process_exact.py
  test_experimental_lightgbm.py
  test_experimental_random_forest.py
  test_experimental_svm.py
Project Total  

This report was generated by python-coverage-comment-action

github-actions[bot] avatar May 21 '25 10:05 github-actions[bot]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 81.48%. Comparing base (71aba97) to head (fcc9357).

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #479      +/-   ##
==========================================
+ Coverage   81.28%   81.48%   +0.20%     
==========================================
  Files         107      107              
  Lines        7420     7503      +83     
==========================================
+ Hits         6031     6114      +83     
  Misses       1389     1389              

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

:rocket: New features to boost your workflow:
  • :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

codecov-commenter avatar May 21 '25 11:05 codecov-commenter