bayesiains
Conditional MADE cannot estimate first dimension of input
Hi, thanks for maintaining this package!
While working on a separate PR in the sbi package, we noticed that MADEs cannot correctly estimate the first dimension of the input when context is not None. After a bit of digging, I think the reason is in the way context is used in MADEs. The output corresponding to the first dimension of the input is always independent of all the inputs (including itself), as enforced by the masking with output_layer=True.
With the current implementation, even if context is provided, any input coming from the hidden layers is masked out in the final layer of MADE, and so the output for the first dimension is simply the bias term of the last layer, regardless of context. I think the simplest fix is to introduce a dummy input variable, as done in this PR. This way the masking constraints are still satisfied between all the real dimensions of the input, and we can retain information from context in the output layer.
Happy to discuss further, and thanks again :)
