edward2
edward2 copied to clipboard
google
bayesian layers
@dustinvtran Hi, I have a question. Can I build bayesian layers with sequential models with lstm using edward2? I mean building sequential model with LSTMCellReparameterization. If yes please help me. I dont know how to build a sequential model with LSTMCellReparameterization layers.
@dustinvtran Please help me. I want to build a sequential model with LSTMCellReparameterization. I really need help.
@dusenberrymw
There's a minimal code snippet in the paper. It should be plug-and-play with any LSTM example you currently have.
Here's a runnable snippet using the reparameterized LSTM cell. As Dustin said, ed.layers.LSTMCellReparameterization is a drop-in replacement for tf.keras.layers.LSTMCell, provided that model.losses is used as part of the loss.
import tensorflow.compat.v2 as tf
import edward2 as ed
num_examples = 2
num_timesteps = 3
input_dim = 4
rnn_dim = 10
inputs = tf.random.normal([num_examples, num_timesteps, input_dim])
labels = tf.random.normal([num_examples])
cell = ed.layers.LSTMCellReparameterization(rnn_dim)
model = tf.keras.Sequential([
tf.keras.layers.RNN(cell),
tf.keras.layers.Dense(1)])
optimizer = tf.keras.optimizers.Adam(0.001)
for i in range(10):
with tf.GradientTape() as tape:
outputs = model(inputs)
nll = tf.reduce_mean((labels-outputs)**2)
kl = sum(model.losses)
loss = nll + kl
grads = tape.gradient(loss, model.variables)
grads_and_vars = zip(grads, model.variables)
optimizer.apply_gradients(grads_and_vars)
print(f"Loss at step {i}: {loss}")
Loss at step 0: 1412.79296875
Loss at step 1: 1412.1939697265625
Loss at step 2: 1411.6484375
Loss at step 3: 1411.123779296875
Loss at step 4: 1410.535888671875
Loss at step 5: 1410.012451171875
Loss at step 6: 1409.4478759765625
Loss at step 7: 1408.9200439453125
Loss at step 8: 1408.29638671875
Loss at step 9: 1407.8123779296875
@dusenberrymw Hi. Thanks a lot for your help. One another question, why the value of loss function is so high in this approach?
@dusenberrymw Hi. I have some questions. What is the benefit of this LSTMCellReparameterization compared to ordinary lstm? Because I think the value of loss function is so high in this approach. And what is the difference between LSTMCellReparameterization and LSTMCellFlipout? How can I calculate accuracy in every epoch?
@dusenberrymw Hi, could you please help me with my questions? I really need them for my master thesis and there isn't so much information about that questions anywhere else. Thanks a lot.
Here's a runnable snippet using the reparameterized LSTM cell. As Dustin said,
ed.layers.LSTMCellReparameterizationis a drop-in replacement fortf.keras.layers.LSTMCell, provided thatmodel.lossesis used as part of the loss.import tensorflow.compat.v2 as tf import edward2 as ed num_examples = 2 num_timesteps = 3 input_dim = 4 rnn_dim = 10 inputs = tf.random.normal([num_examples, num_timesteps, input_dim]) labels = tf.random.normal([num_examples]) cell = ed.layers.LSTMCellReparameterization(rnn_dim) model = tf.keras.Sequential([ tf.keras.layers.RNN(cell), tf.keras.layers.Dense(1)]) optimizer = tf.keras.optimizers.Adam(0.001) for i in range(10): with tf.GradientTape() as tape: outputs = model(inputs) nll = tf.reduce_mean((labels-outputs)**2) kl = sum(model.losses) loss = nll + kl grads = tape.gradient(loss, model.variables) grads_and_vars = zip(grads, model.variables) optimizer.apply_gradients(grads_and_vars) print(f"Loss at step {i}: {loss}")Loss at step 0: 1412.79296875 Loss at step 1: 1412.1939697265625 Loss at step 2: 1411.6484375 Loss at step 3: 1411.123779296875 Loss at step 4: 1410.535888671875 Loss at step 5: 1410.012451171875 Loss at step 6: 1409.4478759765625 Loss at step 7: 1408.9200439453125 Loss at step 8: 1408.29638671875 Loss at step 9: 1407.8123779296875
@dustinvtran Is this running with TF2.0? I'm getting some type error in my code,
lstm_cell = ed.layers.LSTMCellReparameterization(self.lstm_hidden_dim, activation='tanh') lstm_out = tf.keras.layers.RNN(cell=lstm_cell, return_sequences=True)(lstm_in)
{TypeError}Failed to convert object of type <class 'tuple'> to Tensor. Contents: (Dimension(3), 256). Consider casting elements to a supported type.
The error comes from the input tensor lstm_in. I'm using TF1.14, is this causing trouble?
Here's a runnable snippet using the reparameterized LSTM cell. As Dustin said,
ed.layers.LSTMCellReparameterizationis a drop-in replacement fortf.keras.layers.LSTMCell, provided thatmodel.lossesis used as part of the loss.import tensorflow.compat.v2 as tf import edward2 as ed num_examples = 2 num_timesteps = 3 input_dim = 4 rnn_dim = 10 inputs = tf.random.normal([num_examples, num_timesteps, input_dim]) labels = tf.random.normal([num_examples]) cell = ed.layers.LSTMCellReparameterization(rnn_dim) model = tf.keras.Sequential([ tf.keras.layers.RNN(cell), tf.keras.layers.Dense(1)]) optimizer = tf.keras.optimizers.Adam(0.001) for i in range(10): with tf.GradientTape() as tape: outputs = model(inputs) nll = tf.reduce_mean((labels-outputs)**2) kl = sum(model.losses) loss = nll + kl grads = tape.gradient(loss, model.variables) grads_and_vars = zip(grads, model.variables) optimizer.apply_gradients(grads_and_vars) print(f"Loss at step {i}: {loss}")Loss at step 0: 1412.79296875 Loss at step 1: 1412.1939697265625 Loss at step 2: 1411.6484375 Loss at step 3: 1411.123779296875 Loss at step 4: 1410.535888671875 Loss at step 5: 1410.012451171875 Loss at step 6: 1409.4478759765625 Loss at step 7: 1408.9200439453125 Loss at step 8: 1408.29638671875 Loss at step 9: 1407.8123779296875@dustinvtran Is this running with TF2.0? I'm getting some type error in my code,
lstm_cell = ed.layers.LSTMCellReparameterization(self.lstm_hidden_dim, activation='tanh') lstm_out = tf.keras.layers.RNN(cell=lstm_cell, return_sequences=True)(lstm_in)
{TypeError}Failed to convert object of type <class 'tuple'> to Tensor. Contents: (Dimension(3), 256). Consider casting elements to a supported type.
The error comes from the input tensor lstm_in. I'm using TF1.14, is this causing trouble?
@dustinvtran I upgraded everything to TF2.0 and it works. However, I still want to ask if there is a way you could let me use TF1.14 with the lstmReparameterization layer? Like modify the source code somewhere?