FractalNet: Ultra-deep neural networks without residuals

PyTorch Implementation of FractalNet.

Larsson, Gustav, Michael Maire, and Gregory Shakhnarovich. "Fractalnet: Ultra-deep neural networks without residuals." arXiv preprint arXiv:1605.07648 (2016).

Requirements

pip install -r requirements.txt

• python 3
• pytorch >= 0.4
• torchvision
• numpy, tensorboard, tensorboardX

Usage

Default options are same as the paper, but I failed to reproduce the results of paper with the default options. So I've tried a lot to improve the performance and found a better option. best indicates this option.

# train (default)
python train.py --data cifar10 --name cifar10
# train (best)
python train.py --data cifar10 --name cifar10-best --init torch --gap 1 --pad reflect

For the test, you must specify the same name and same options that you used for the training. Some options are only required for training such as epochs, dropout, droppath, init, and data augmentation.

# test (for deepest)
python test.py --data cifar10 --name cifar10
python test.py --data cifar10 --name cifar10-best --init torch --gap 1 --pad reflect

Run options

Train

$ python train.py --help
usage: Config [-h] --name NAME [--data DATA] [--batch_size BATCH_SIZE]
              [--lr LR] [--momentum MOMENTUM] [--print_freq PRINT_FREQ]
              [--gpu GPU] [--epochs EPOCHS] [--init_channels INIT_CHANNELS]
              [--gdrop_ratio GDROP_RATIO] [--p_ldrop P_LDROP]
              [--dropout_probs DROPOUT_PROBS] [--blocks BLOCKS]
              [--columns COLUMNS] [--seed SEED] [--workers WORKERS]
              [--aug_lv AUG_LV] [--off-drops] [--gap GAP] [--init INIT]
              [--pad PAD] [--doubling] [--gdrop_type GDROP_TYPE]
              [--dropout_pos DROPOUT_POS]

optional arguments:
  -h, --help            show this help message and exit
  --name NAME
  --data DATA           CIFAR10 (default) / CIFAR100
  --batch_size BATCH_SIZE
                        default: 100
  --lr LR               learning rate (default: 0.02)
  --momentum MOMENTUM
  --print_freq PRINT_FREQ
                        print frequency
  --gpu GPU             gpu device id
  --epochs EPOCHS       # of training epochs (default: 400)
  --init_channels INIT_CHANNELS
                        doubling each block except the last (default: 64)
  --gdrop_ratio GDROP_RATIO
                        ratio of global drop path (default: 0.5)
  --p_ldrop P_LDROP     local drop path probability (default: 0.15)
  --dropout_probs DROPOUT_PROBS
                        dropout probs for each block with comma separated
                        (default: 0.0, 0.1, 0.2, 0.3, 0.4)
  --blocks BLOCKS       default: 5
  --columns COLUMNS     default: 3
  --seed SEED           random seed
  --workers WORKERS     # of workers
  --aug_lv AUG_LV       data augmentation level (0~2). 0: no augmentation, 1:
                        horizontal mirroring + [-4, 4] translation, 2: 1 +
                        cutout.

Experiment:
  --off-drops           turn off all dropout and droppath
  --gap GAP             0: max-pool (default), 1: GAP - FC, 2: 1x1conv - GAP
  --init INIT           xavier (default) / he / torch (pytorch default)
  --pad PAD             zero (default) / reflect
  --doubling            turn on 1x1 conv channel doubling
  --gdrop_type GDROP_TYPE
                        ps (per-sample, various gdrop per block) / ps-consist
                        (default; per-sample, consist global drop)
  --dropout_pos DROPOUT_POS
                        CDBR (default; conv-dropout-BN-relu) / CBRD (conv-BN-
                        relu-dropout) / FD (fractal_block-dropout)

Test

$ python test.py --help
usage: Config [-h] --name NAME [--data DATA] [--batch_size BATCH_SIZE]
              [--print_freq PRINT_FREQ] [--gpu GPU]
              [--init_channels INIT_CHANNELS] [--blocks BLOCKS]
              [--columns COLUMNS] [--workers WORKERS] [--gap GAP] [--pad PAD]
              [--doubling] [--dropout_pos DROPOUT_POS]

optional arguments:
  -h, --help            show this help message and exit
  --name NAME
  --data DATA           CIFAR10 (default) / CIFAR100
  --batch_size BATCH_SIZE
                        default: 200
  --gpu GPU             gpu device id
  --init_channels INIT_CHANNELS
                        doubling each block except the last (default: 64)
  --blocks BLOCKS       default: 5
  --columns COLUMNS     default: 3
  --workers WORKERS     # of workers

Experiment:
  --gap GAP             0: max-pool (default), 1: GAP - FC, 2: 1x1conv - GAP
  --pad PAD             zero (default) / reflect
  --doubling            turn on 1x1 conv channel doubling

Results

Disclaimer

• The data augmentation method for C10++ is not described in the paper. Here I used the Cutout method in addition to horizontal mirroring and [-4, 4] translation.
• In C10++ and C100++, the epoch was increased to 600, if necessary.

Best plots

Best results for CIFAR10 and CIFAR100. 3.91% and 21.64%, respectively.

CIFAR10

As mentioned before, the results of the paper were not reproduced. After several attempts, I've got the best option, which is --init torch --gap 1 --pad reflect. The best option got about 1.2% better than the default.

Furthermore, I got better results with less regularization. The last row is that: "Best + FDO (Fractal dropout) + local DP (No global drop-path)". The run option is --init torch --gap 1 --pad reflect --global_drop_ratio 0. --dropout_pos FD.

CIFAR100

Likewise C100, the paper results were not reproduced. In C100, I did not perform as many experiments as C10. But the results is similar - "Best + FDO + local DP" is better.

However, there are some difference: default option is better than best option in C100+ and C100++, and the doubling works better in C100 than in C10.

ETC

• The Keras implementation snf/keras-fractalnet also failed to reproduce the results of the paper.
• If you are familiar with Korean, there are more discussions and results in exp-note (kor).