the issue regrading clockwork_mask (the upper triangular matrix)

[dzhu622911]

According to the [arXiv, https://arxiv.org/abs/1402.3511], My understanding is the mask matrix shape is [num_group * group_size, num_group * group_size], can also image the mask is constructed by [num_group, num_group] but each element in matrix is a sub-matrix which shape is [group_size, group_size];

so i think the expected clockwork_mask is upper triangular matrix of [num_group, num_group], but not [num_group * group_size, num_group * group_size];

i also modify the code, change the below code:

self.clockwork_mask = tf.constant(np.triu(np.ones([self.config.num_hidden, self.config.num_hidden])), dtype=tf.float32, name="mask")

to,

        mask = np.zeros((self.config.num_hidden, self.config.num_hidden))
        for i in range(len(self.clockwork_periods)):
            mask[i*self.group_size:(i+1)*self.group_size, i*self.group_size:] = 1
        self.clockwork_mask = tf.constant(mask, name="mask")

[dzhu622911]

sorry, should be this:

        mask = np.zeros((self.config.num_hidden, self.config.num_hidden))
        for i in range(len(self.clockwork_periods)):
            mask[i*self.group_size:(i+1)*self.group_size, i*self.group_size:] = 1
        self.clockwork_mask = tf.constant(mask, dtype=tf.float32, name="mask")