Bias shape of Conv2D

[t-ae]

I found strange use of Conv2D in its test. https://github.com/tensorflow/swift-apis/blob/3304db3e728120b55674cca06894b6ea5083b5e8/Tests/TensorFlowTests/LayerTests.swift#L101-L111

The filter has shape [1, 2, 2, 1] and input has shape [2, 2, 2, 2]. Conv2D has strides (2, 2), so output of internal conv2D will have shape [2, 1, 1, 1]. But the bias has shape [2]. Therefore final output will be broadcasted shape [2, 1, 1, 2].

Currently S4TF is allowing bias to be any shape (as long as it can be added to conv2D output). But Keras, for example, restricts it to have shape [filter.shape[3]]. So do PyTroch.

Are there any good at allowing free shape biases? I think it's better to ban it and add precondition in initializer:

precondition(filter.shape[3] == bias.shape[0],
             "The number of output features doesn't match between filter and bias.")

In this way we don't have to check if we can add conv2D output and bias in callAsFunction. (If the bias shape is free, we can't check beforehand because we need input shape for it.) Even if some user wants to use bias that has different shape, he can do it by simply having another tensor outside the layer.

Also there's same issue for Dense, *Conv*D.

[Shashi456]

I think we sort of already do what you mentioned. Whenever the initializer is used the bias size is restricted to filter.shape[filter.ndim-1], as you can see here and here, similarly for other layers you mentioned about. @sgugger added the useBias functionality by which we only initialize learnable bias whenever needed.

The test was written in mind for testing the api thoroughly, so I didn't exactly look at bias shape checks.

We can explicitly have shape checks for bias, since usually the bias is applied per-channel, so the number of weights (if non-zero) must be equal to the number of output feature maps but i don't think its necessary.