[WIP] added PyTorch Profiler

[Ishan-Kumar2]

Fixes #1917

Description: Added a minimum implementation of PyTorch profiler as a handler with engine. A lot of other features can be added, please let me know if you have any suggestions. Also I haven't added tests yet, if the initial code looks good, I can get started on those :)

Check list:

• [ ] New tests are added (if a new feature is added)
• [X] New doc strings: description and/or example code are in RST format
• [X] Documentation is updated (if required)

[sdesrozis]

@Ishan-Kumar2 Thank you ! It looks great, I will play with the handler asap !

[vfdev-5]

@sdesrozis @Ishan-Kumar2 can we move on with this PR ?

[Ishan-Kumar2]

Hi @vfdev-5, sorry for the delay. I'll have a look at it today :)

[sdesrozis]

@Ishan-Kumar2 Sorry for the delay on my side. I will use this handler in some own codes to see whether it works and give a feedback asap.

[Ishan-Kumar2]

@sdesrozis I tested the code on my local example, with this additional code

# Define a PT Profiler
pt_profiler = PyTorchProfiler(on_trace_ready="tensorboard", output_path="./logs/train")
pt_profiler.attach(trainer)

it produces 3 json files (non empty), but I am unable to load them on tensorboard using

tensorboard --logdir=./logs

it shows "No dashboards are active for the current data set." I think as long as json are being produced it should be correct since that's done by PyTorch Profiler, not changed by me. There must be some issue with my opening it. Not sure what is causing this, if you have a colab example could you please share the snippet you use to install ignite from this PR. I'll check running on colab too.

[sdesrozis]

@sdesrozis I tested the code on my local example, with this additional code

# Define a PT Profiler
pt_profiler = PyTorchProfiler(on_trace_ready="tensorboard", output_path="./logs/train")
pt_profiler.attach(trainer)

it produces 3 json files (non empty), but I am unable to load them on tensorboard using

tensorboard --logdir=./logs

it shows "No dashboards are active for the current data set." I think as long as json are being produced it should be correct since that's done by PyTorch Profiler, not changed by me. There must be some issue with my opening it. Not sure what is causing this, if you have a colab example could you please share the snippet you use to install ignite from this PR. I'll check running on colab too.

It works for me but don't forget to install the dedicated tensorboard pluggin

pip install torch_tb_profiler

I left a few comments.

[sdesrozis]

@Ishan-Kumar2 it looks good. I think the next step now is about the tests.

[Ishan-Kumar2]

@sdesrozis great, will start working on the tests.

[Ishan-Kumar2]

@sdesrozis, Added some tests for the profiler. I have not added checks for the output of the profiler since I believe that is already done by PyTorch. I am new to writing tests from scratch so please let me know if I need to tests something else. Thanks!

[sdesrozis]

@sdesrozis, Added some tests for the profiler. I have not added checks for the output of the profiler since I believe that is already done by PyTorch. I am new to writing tests from scratch so please let me know if I need to tests something else. Thanks!

You can get inspiration from these tests tests/ignite/contrib/handlers/test_tensorboard_logger.py.

The tests need to be improved to check what is going on with the different backends (tpu, nccl, etc).

[Ishan-Kumar2]

@sdesrozis I have incorporated most of your suggestions. I am still working on the distributed tests will add those soon too.