sagemaker-debugger icon indicating copy to clipboard operation
sagemaker-debugger copied to clipboard

Extended Debugger reductions and fixed some bugs

Open NRauschmayr opened this issue 4 years ago • 1 comments

Description of changes:

Extended smdebug's reductions to check for nan- and inf-values and to compute quantiles for PT tensors. Tensors are now also written out in Tensorboard format such that users can display all reductions for a specific tensor within the same visualization and visualizations will be grouped by Debugger collections. Here is an example visualization:

Screen Shot 2021-11-14 at 2 52 42 PM

Style and formatting:

I have run pre-commit install && pre-commit run --all-files to ensure that auto-formatting happens with every commit.

Issue number, if available

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

NRauschmayr avatar Nov 14 '21 22:11 NRauschmayr

Codecov Report

Merging #523 (66999aa) into master (b4dd4c1) will decrease coverage by 6.36%. The diff coverage is 57.44%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #523      +/-   ##
==========================================
- Coverage   77.60%   71.24%   -6.37%     
==========================================
  Files         127      117      -10     
  Lines       11111    10614     -497     
==========================================
- Hits         8623     7562    -1061     
- Misses       2488     3052     +564     
Impacted Files Coverage Δ
smdebug/pytorch/utils.py 48.14% <23.52%> (-32.81%) :arrow_down:
smdebug/core/locations.py 85.71% <60.00%> (-5.96%) :arrow_down:
smdebug/core/reduction_config.py 86.58% <77.77%> (-9.52%) :arrow_down:
smdebug/core/hook.py 86.90% <81.25%> (-0.53%) :arrow_down:
smdebug/mxnet/__init__.py 0.00% <0.00%> (-100.00%) :arrow_down:
smdebug/mxnet/singleton_utils.py 0.00% <0.00%> (-100.00%) :arrow_down:
...debug/profiler/analysis/notebook_utils/__init__.py 0.00% <0.00%> (-100.00%) :arrow_down:
smdebug/mxnet/hook.py 0.00% <0.00%> (-84.85%) :arrow_down:
smdebug/mxnet/utils.py 0.00% <0.00%> (-78.13%) :arrow_down:
smdebug/rules/action/message_action.py 13.25% <0.00%> (-75.91%) :arrow_down:
... and 61 more

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update b4dd4c1...66999aa. Read the comment docs.

codecov-commenter avatar Nov 14 '21 23:11 codecov-commenter