Ray.tune for parameter optimization. Included more functionality

[nrclaudio]

Hi @adamgayoso and @PierreBoyeau. As we discussed recently via email, in our lab we were able to use the logic in michael/autotune to do hyperparameter optimization.

I've implemented the changes I applied to the original Autotune module. I also included the option to test for different number of HVGs, use of covariates in the model HVG filters. In our implementation we also included a descriptive plot for the tune experiment. I didn't include this in the module but we can discuss if it's something worth including.

The environment needs to have ray tune installed with pip install -U "ray[tune]" but I wasn't sure how to add dependencies to the PR.

*y ax: silhouette score *x ax: epoch

I'm not sure if I should've PRd to master instead, let me know if that's the case and I can open a new one there.

[justjhong]

Hi @nrclaudio, thanks for providing this PR. If you could PR to master instead that would be great. Right now the changes include everything on master since the original autotune branch which makes it hard to parse out the contributed code.

[nrclaudio]

Hi @jjhong922, I think it should be up to date now

[nrclaudio]

Thanks for the update on this PR and the extensive testing! To add ray tune as a dependency, we would want to add it to the pyproject.toml file as an optional dependency, then create a new extras keyword tune under which you put ray. You can see how pymde was handled in that file as an example.

I'll let @adamgayoso or @PierreBoyeau take a pass as well.

I've tried changing pyproject.toml, but it seems that the job it's not installing tune's dependencies during pytest. Let me know if I did something wrong there

[justjhong]

@nrclaudio adding tune to the list of package extras in this line should fix that https://github.com/scverse/scvi-tools/blob/master/.github/workflows/test.yml#L36

[codecov[bot]]

Codecov Report

Merging #1565 (522395e) into master (a019d9d) will decrease coverage by 0.47%. The diff coverage is 69.56%.

@@            Coverage Diff             @@
##           master    #1565      +/-   ##
==========================================
- Coverage   91.23%   90.75%   -0.48%     
==========================================
  Files         116      122       +6     
  Lines        9196     9403     +207     
==========================================
+ Hits         8390     8534     +144     
- Misses        806      869      +63     

Impacted Files	Coverage Δ
scvi/autotune/_metrics.py	20.68% <20.68%> (ø)
scvi/autotune/_callbacks.py	32.50% <32.50%> (ø)
scvi/autotune/_utils.py	83.33% <83.33%> (ø)
scvi/autotune/_autotune.py	91.56% <91.56%> (ø)
scvi/autotune/__init__.py	100.00% <100.00%> (ø)
scvi/autotune/_wrappers.py	100.00% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update a019d9d...522395e. Read the comment docs.

[justjhong]

We'd like to retain the commit history for the original ray tune PR. Hopefully this isn't too much work, but could you try rebasing on michael/autotune and updating this PR? You wouldn't change the base since we would just merge the commits together in one PR to master. I just updated the michael/autotune PR to latest master, but tests are failing since I didn't adapt it to the new setup scheme which you did in this PR.

[nrclaudio]

I see @adamgayoso included autotune.srt already for documentation, I can help draft it if you want.

[justjhong]

Also, please resolve the codacy checks if you can

[justjhong]

Also the docs are still broken, that should be fixed before merging

[adamgayoso]

@nrclaudio we will merge this into a staging branch and complete it on our end. Thank you for the contribution!!