Efficient Deep Learning

1.Introduction

• With the rapid development of deep learning, more and more people are flocking to this field, including me. As a former rookie, I have experienced various problems during my deep learning process, so I create this repo here to record some tricks that can make you have an efficient deep learning. You are also welcome to raise a PR and give some of your tips!

2.Efficient Coding

• Strategies to code efficiently.
• Efficient Coding
  • Use Vscode
  • Auto code formating
  • Pre-commit hook
  • Learn to use git
  • Grammarly
  • StackOverflow
  • Auto docstring

3.Efficient Data Processing

• Strategies to speed up your data processing.
• Efficient Data Processing
  • SSD
  • num_workers and pin_memory
  • LMDB file
  • Albumentations
  • Data augmentation on GPU

4.Efficient Training

• Strategies to speed up your training process.
• Efficient Traininig
  • cudnn.benchmark=True
  • Set gradients to None during back propagation
  • Turn off debugging APIs
  • Turn off gradient computation during validation
  • Use another optimizer AdamW
  • Learning rate schedule
  • Useful combination, Adam with 3e-4
  • LR Warm up and Cosine Learning Rate Decay
  • L2 decay

5.Efficient GPUtilization

• Strategies to have a better GPU utilization.
• Efficient GPUtilization
  • CUDA out of memory solutions
  • Automatic Mixed Precision (AMP)
  • Gradient Accumulation
  • Gradient Checkpoint
  • Data parallelization training

6.Efficient Tools

• A list of useful tools.
• Efficient Tools
  • Torchinfo: Visualize Network Architecture
  • drawio: Free graphing software
  • Octotree: Free gitHub code tree
  • ACRONYMIFY: Name your paper with a cool acronyms
  • Linggle: Grammer checker
  • AI pair programmer: Github Copilot

• "The past decade has seen tremendous progress in the field of artificial intelligence thanks to the resurgence of neural networks through deep learning. This has helped improve the ability for computers to see, hear, and understand the world around them, leading to dramatic advances in the application of AI to many fields of science and other areas of human endeavor" ——Jeffrey Dean