Implementing options in various Layers

[Shashi456]

Currently, we define all layers assuming everything to be true by default. For example, bias within a variety of layers could be added as a boolean option, and initialization of the recurrent layers is done by glorotUniform, when more initializations are added, we might also want to offer that flexibility to the user.

Similarly within convolution layers and other layers that are present and are yet to be added might benefit from this added flexibility.

I think we could also maybe have a discussion of whether providing an option makes sense in a few cases, for example if bias is made boolean, then some part of the code becomes redundant.

if bias == true :
    b_ih = Tensor( glorotUniform : shape, seed)
.
.
earlier : 
op = tanh(weight + bias)

now : 
op = weight 
if bias = true:
     op + = bias 
output = tanh(op)

Sometimes options make the code repetitive and bloated, So i'm wondering if there is any work to around this.

#38 also talks about this problem in similar lines.

[tanmayb123]

I'd say that it's great to have it as an option & that if statements would be the best way to do this. With autodiff supporting control flow, a lot of redundancy will be reduced.

[dan-zheng]

Conditionally conforming Optional to Differentiable may be a good first step.

// stdlib/public/core/Optional.swift
extension Optional : Differentiable where Wrapper : Differentiable { ... }

TF-365 tracks this. Please self-assign the issue if you're interested in tackling this!

[Shashi456]

@dan-zheng I would like to parallelly work on adding this. Could you tell me how i could go about conforming Optional to Differentiable?