axon
axon copied to clipboard
Add weight sharing
There is currently no explicit way to use the same weights/layer for separate inputs. For example, in Keras I could do:
dense = tf.keras.layers.Dense(32)
d_x1 = dense(x1)
d_x2 = dense(x2)
And it would use the same layer for both operations. I believe this could potentially work for now:
x1 = Axon.input({nil, 32})
x2 = Axon.input({nil, 32})
d_x1 = Axon.dense(x1, 64, name: "dense")
d_x2 = Axon.dense(x2, 64, name: "dense")
And the parameters would be shared by name. Perhaps there is a way to make this a bit more explicit. This somewhat relates to some of the issues that appeared earlier with GANs and the proposed Axon.function
Maybe an alternative is to accept a params option with the name of a previous layer as a value.
With the introduction of namespaces and the new custom layer API there are a few additional open questions:
- How do we deal with duplicate usage of the same param (it will have the same ID) in two separate custom layers. To me the parameters should be tied, it is the same parameter!
- How do we deal with duplicate usage of the same namespace? If I define two namespaces (with different layers), and then give them the same name, what should be the intended behavior?