Neural network meta-issue

[DilumAluthge]

This is a meta-issue to track the issues that we need to solve in order for Octavian to become a competitive option for training neural networks on the CPU.

• [ ] #40
• [ ] #56

[DilumAluthge]

What kinds of layers do we need to start with?

1. Dense (fully connected) layer
2. Convolutional layer
3. Max pooling layer
4. Average pooling layer

That should be sufficient to e.g. do LeNet-5, right?

@chriselrod

[chriselrod]

Yeah, from the model zoo:

# LeNet5 "constructor". 
# The model can be adapted to any image size
# and any number of output classes.
function LeNet5(; imgsize=(28,28,1), nclasses=10) 
    out_conv_size = (imgsize[1]÷4 - 3, imgsize[2]÷4 - 3, 16)
    
    return Chain(
            Conv((5, 5), imgsize[end]=>6, relu),
            MaxPool((2, 2)),
            Conv((5, 5), 6=>16, relu),
            MaxPool((2, 2)),
            flatten,
            Dense(prod(out_conv_size), 120, relu), 
            Dense(120, 84, relu), 
            Dense(84, nclasses)
          )
end