Single Layer Encoder

[blester125]

In your code it seems like you are trying to create a multilayer encoder, however what is actually happening is that multiple single layer decoders are being created. As you can see in this graph image created by tensorflow http://imgur.com/a/jqAn5 the rnn inputs are feed into each bidirectional rnn rather than one feeding into the other. I created this image by running your encoding_layer function and passing in some placeholders and then using the tf.summary.FileWriter to draw the graph.

To create a multilayer encoder you would want something like this

def encoding_layer(rnn_size, sequence_length, num_layers, rnn_inputs, keep_prob):
    '''Create the encoding layer'''
    layer_input = rnn_inputs
    for layer in range(num_layers):
        with tf.variable_scope('encoder_{}'.format(layer)):
            cell_fw = tf.contrib.rnn.LSTMCell(rnn_size,
                                              initializer=tf.random_uniform_initializer(-0.1, 0.1, seed=2))
            cell_fw = tf.contrib.rnn.DropoutWrapper(cell_fw, 
                                                    input_keep_prob = keep_prob)

            cell_bw = tf.contrib.rnn.LSTMCell(rnn_size,
                                              initializer=tf.random_uniform_initializer(-0.1, 0.1, seed=2))
            cell_bw = tf.contrib.rnn.DropoutWrapper(cell_bw, 
                                                    input_keep_prob = keep_prob)

            enc_output, enc_state = tf.nn.bidirectional_dynamic_rnn(cell_fw, 
                                                                    cell_bw, 
                                                                    layer_input,
                                                                    sequence_length,
                                                                    dtype=tf.float32)
            layer_input = tf.concat(enc_output, 2)
    # Join outputs since we are using a bidirectional RNN
    enc_output = tf.concat(enc_output,2)
    return enc_output, enc_state

You can see in this graph image http://imgur.com/a/bJANa That this creates the multi layer structure I assume you are going for. (The caveat is that in this function your rnn size must be half of the embedding size so if you want them to be different just move the first birnn out of the loop and have the layers after the first have the same size)

[johnsonice]

@blester125 yes, you are right. i was really confused when i am reading author's code