dlrover icon indicating copy to clipboard operation
dlrover copied to clipboard

Torch Trainer Hook

Open Antlera opened this issue 2 years ago • 2 comments

For this issue, the objective is to create a hook or callback system in our PyTorch trainer that would allow it to invoke resource monitoring and time reporting at the start of training. This hook should be well-integrated into the training process and should not interfere with the main training tasks. We need to design this hook in a way that it can trigger our resources reporter, and potentially, other types of monitors we may add in the future. We can implement this manually or use callback mechanisms similar to what is available in PyTorch Lightning.

Antlera avatar Jul 28 '23 14:07 Antlera

We can support the Trainer in lighting and implement a lighting callback

workingloong avatar Jul 29 '23 11:07 workingloong

We can support the Trainer in lighting and implement a lighting callback

Thank you for your prompt and helpful response! I'll definitely look into implementing the lighting callback as you suggested. Your provided link will be a valuable resource for my implementation. Thanks again!

Antlera avatar Jul 29 '23 12:07 Antlera

This issue has been automatically marked as stale because it has not had recent activity.

github-actions[bot] avatar Oct 27 '24 01:10 github-actions[bot]