convnetjs icon indicating copy to clipboard operation
convnetjs copied to clipboard

[ConvNetJS] Implement DropConnect for regularizing large fully-connected layers

Open ajtulloch opened this issue 10 years ago • 0 comments

Summary

This diff implements DropConnect, following the paper in http://cs.nyu.edu/~wanli/dropc/dropc.pdf.

"When training with Dropout, a randomly selected subset of activations are set to zero within each layer. DropConnect instead sets a randomly selected subset of weights within the network to zero. Each unit thus receives input from a random subset of units in the previous layer."

DropConnect has been shown to achieve significant improvements in model performance over DropOut.

Test Plan

I'm not sure on how to proceed here - do you want me to add a demo demonstrating DropOut vs DropConnect on an otherwise identical dataset/architecture?

I'd like to add unit tests but it's unclear how to add them into this project.

I verified that existing demonstrations (MNIST, autoencoder, etc) work, and that adding a drop-connect layer allows training to continue as expected.

ajtulloch avatar Apr 19 '14 09:04 ajtulloch