satpy icon indicating copy to clipboard operation
satpy copied to clipboard
verified publisher icon pytroll

Add "performance_test" to the benchmarks

Open yukaribbba opened this issue 1 year ago β€’ 25 comments

To have a better understanding of the performance topics in FAQ, several tests are taken. This PR will give brief on them.

  • [x] Fully documented

yukaribbba avatar Jun 30 '24 03:06 yukaribbba

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 95.96%. Comparing base (440925d) to head (0129fdd). Report is 1396 commits behind head on main.

Additional details and impacted files 
@@           Coverage Diff           @@
##             main    #2847   +/-   ##
=======================================
  Coverage   95.96%   95.96%           
=======================================
  Files         366      366           
  Lines       53593    53619   +26     
=======================================
+ Hits        51433    51458   +25     
- Misses       2160     2161    +1
Flag Coverage Ξ”
behaviourtests 4.04% <ΓΈ> (-0.01%) :arrow_down:
unittests 96.06% <ΓΈ> (+<0.01%) :arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

:rocket: New features to boost your workflow:
  • :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

codecov[bot] avatar Jun 30 '24 03:06 codecov[bot]

Pull Request Test Coverage Report for Build 9729113408

Details

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage remained the same at 96.046%
Totals Coverage Status
Change from base Build 9702977764: 0.0%
Covered Lines: 51641
Relevant Lines: 53767
πŸ’› - Coveralls

coveralls avatar Jun 30 '24 04:06 coveralls

Pull Request Test Coverage Report for Build 9729246492

Details

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage remained the same at 96.046%
Totals Coverage Status
Change from base Build 9702977764: 0.0%
Covered Lines: 51641
Relevant Lines: 53767
πŸ’› - Coveralls

coveralls avatar Jun 30 '24 04:06 coveralls

Pull Request Test Coverage Report for Build 9730630372

Details

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage remained the same at 96.046%
Totals Coverage Status
Change from base Build 9702977764: 0.0%
Covered Lines: 51641
Relevant Lines: 53767
πŸ’› - Coveralls

coveralls avatar Jun 30 '24 08:06 coveralls

Pull Request Test Coverage Report for Build 9733225497

Details

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage remained the same at 96.046%
Totals Coverage Status
Change from base Build 9702977764: 0.0%
Covered Lines: 51641
Relevant Lines: 53767
πŸ’› - Coveralls

coveralls avatar Jun 30 '24 16:06 coveralls

Pull Request Test Coverage Report for Build 9733424593

Details

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage remained the same at 96.046%
Totals Coverage Status
Change from base Build 9702977764: 0.0%
Covered Lines: 51641
Relevant Lines: 53767
πŸ’› - Coveralls

coveralls avatar Jun 30 '24 16:06 coveralls

Pull Request Test Coverage Report for Build 9747623796

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

Details

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage remained the same at 96.046%
Totals Coverage Status
Change from base Build 9702977764: 0.0%
Covered Lines: 51641
Relevant Lines: 53767
πŸ’› - Coveralls

coveralls avatar Jul 01 '24 16:07 coveralls

I agree. Actually I'm also not sure how useful my results could be for other platforms (That's why I only post one of them). I do have the test scripts that can finally output a csv file just like the tables in this PR. I'm making some improvements for public use. Maybe we can just let the user run it and decide what's the best settings like you said. But where shall we put it?

yukaribbba avatar Jul 02 '24 02:07 yukaribbba

Iβ€˜ve made some changes. Now we can review the html report of the test. FireShot Capture 004 - Satpy Performance Test Report for abi_l1b -

yukaribbba avatar Jul 04 '24 12:07 yukaribbba

pre-commit.ci autofix

yukaribbba avatar Jul 07 '24 01:07 yukaribbba

Pull Request Test Coverage Report for Build 9823636005

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

Details

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • 9 unchanged lines in 2 files lost coverage.
  • Overall coverage increased (+0.02%) to 96.067%
Files with Coverage Reduction New Missed Lines %
satpy/tests/reader_tests/gms/test_gms5_vissr_navigation.py 1 99.3%
satpy/tests/utils.py 8 94.02%
<!-- Total: 9
Totals Coverage Status
Change from base Build 9702977764: 0.02%
Covered Lines: 51660
Relevant Lines: 53775
πŸ’› - Coveralls

coveralls avatar Jul 07 '24 02:07 coveralls

pre-commit.ci autofix

yukaribbba avatar Jul 07 '24 03:07 yukaribbba

Pull Request Test Coverage Report for Build 9824374171

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

Details

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • 11 unchanged lines in 2 files lost coverage.
  • Overall coverage increased (+0.006%) to 96.052%
Files with Coverage Reduction New Missed Lines %
satpy/tests/reader_tests/gms/test_gms5_vissr_navigation.py 1 97.18%
satpy/tests/utils.py 10 93.16%
<!-- Total: 11
Totals Coverage Status
Change from base Build 9702977764: 0.006%
Covered Lines: 51652
Relevant Lines: 53775
πŸ’› - Coveralls

coveralls avatar Jul 07 '24 04:07 coveralls

Pull Request Test Coverage Report for Build 9825388854

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

Details

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • 11 unchanged lines in 2 files lost coverage.
  • Overall coverage increased (+0.006%) to 96.052%
Files with Coverage Reduction New Missed Lines %
satpy/tests/reader_tests/gms/test_gms5_vissr_navigation.py 1 97.18%
satpy/tests/utils.py 10 93.16%
<!-- Total: 11
Totals Coverage Status
Change from base Build 9702977764: 0.006%
Covered Lines: 51652
Relevant Lines: 53775
πŸ’› - Coveralls

coveralls avatar Jul 07 '24 07:07 coveralls

Pull Request Test Coverage Report for Build 9826570361

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

Details

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • 11 unchanged lines in 2 files lost coverage.
  • Overall coverage increased (+0.006%) to 96.052%
Files with Coverage Reduction New Missed Lines %
satpy/tests/reader_tests/gms/test_gms5_vissr_navigation.py 1 97.18%
satpy/tests/utils.py 10 93.16%
<!-- Total: 11
Totals Coverage Status
Change from base Build 9702977764: 0.006%
Covered Lines: 51652
Relevant Lines: 53775
πŸ’› - Coveralls

coveralls avatar Jul 07 '24 10:07 coveralls

Pull Request Test Coverage Report for Build 9827365298

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

Details

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • 11 unchanged lines in 2 files lost coverage.
  • Overall coverage increased (+0.006%) to 96.052%
Files with Coverage Reduction New Missed Lines %
satpy/tests/reader_tests/gms/test_gms5_vissr_navigation.py 1 97.18%
satpy/tests/utils.py 10 93.16%
<!-- Total: 11
Totals Coverage Status
Change from base Build 9702977764: 0.006%
Covered Lines: 51652
Relevant Lines: 53775
πŸ’› - Coveralls

coveralls avatar Jul 07 '24 13:07 coveralls

Pull Request Test Coverage Report for Build 9827748182

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

Details

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • 11 unchanged lines in 2 files lost coverage.
  • Overall coverage increased (+0.006%) to 96.052%
Files with Coverage Reduction New Missed Lines %
satpy/tests/reader_tests/gms/test_gms5_vissr_navigation.py 1 97.18%
satpy/tests/utils.py 10 93.16%
<!-- Total: 11
Totals Coverage Status
Change from base Build 9702977764: 0.006%
Covered Lines: 51652
Relevant Lines: 53775
πŸ’› - Coveralls

coveralls avatar Jul 07 '24 14:07 coveralls

Pull Request Test Coverage Report for Build 9828713154

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

Details

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • 11 unchanged lines in 2 files lost coverage.
  • Overall coverage increased (+0.006%) to 96.052%
Files with Coverage Reduction New Missed Lines %
satpy/tests/reader_tests/gms/test_gms5_vissr_navigation.py 1 97.18%
satpy/tests/utils.py 10 93.16%
<!-- Total: 11
Totals Coverage Status
Change from base Build 9702977764: 0.006%
Covered Lines: 51652
Relevant Lines: 53775
πŸ’› - Coveralls

coveralls avatar Jul 07 '24 16:07 coveralls

@djhoese CI failed again. Anyway my part is done temporarily. Here's an example of the test report. FireShot Capture 005 - Satpy Performance Test Report for abi_l1b -

yukaribbba avatar Jul 08 '24 04:07 yukaribbba

Pull Request Test Coverage Report for Build 9853151991

Details

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage remained the same at 96.069%
Totals Coverage Status
Change from base Build 9842330086: 0.0%
Covered Lines: 51687
Relevant Lines: 53802
πŸ’› - Coveralls

coveralls avatar Jul 09 '24 08:07 coveralls

I did think about command line options but eventually I found it looks nasty to let users put all these options in command. Beside, there're some kwargs hard to input that way. But ok, I'll be waiting for others' thoughts.

yukaribbba avatar Jul 09 '24 15:07 yukaribbba

For dependencies, we can drop py-cpuinfo but psutil is necessary. Otherwise how can we record cpu/memory usage?

yukaribbba avatar Jul 09 '24 15:07 yukaribbba

For dependencies, we can drop py-cpuinfo but psutil is necessary. Otherwise how can we record cpu/memory usage?

Basically I was thinking that the script/module wouldn't be added to the docs (no API docs) and therefore we wouldn't need any of its dependencies because sphinx would never import it to be documented. Alternatively, we could tell sphinx to mock its dependencies in conf.py.

djhoese avatar Jul 09 '24 15:07 djhoese

Well for me reading the docs are always better than reading the description inside the codes. Thats why I embed it. But the doc thing is just secondary. First we gonna hear what others say.

yukaribbba avatar Jul 09 '24 16:07 yukaribbba

Sorry for being so late on this. First I'd like to thank you @yukaribbba for working on this! As a user, I can imaging the script being really useful when you want to know what parameters to tweak for your particular data.

A couple of comments/questions (I haven't checked the code in detail yet, so forgive me if I overlooked something):

  • for read the docs, you don't actually need matplotlib I think, if you use autodoc_mock_imports.
  • Have you considered including dask graphs also?
  • For other satpy benchmarks, we are using airspeed velocity and showing the results here: https://mraspaud.github.io/satpy/ (hasn't been updated for a year though :scream: , I need to fix that). Would this be something we could use ? It uses a VM on the European weather cloud to do the benchmarking.

mraspaud avatar Jul 26 '24 08:07 mraspaud