About norm

[SongFGH]

Hi @tkipf: I have question about norm in 78 line in train.py https://github.com/tkipf/gae/blob/master/gae/train.py So ,can you explain it?

[tkipf]

Yes, have a look here: https://arxiv.org/abs/1609.02907

On Mon 3. Dec 2018 at 06:09 Long Wu [email protected] wrote:

Hi @tkipf https://github.com/tkipf: I have question about norm in train.py https://github.com/tkipf/gae/blob/master/gae/train.py http://url. So ,can you explain it?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/tkipf/gae/issues/17, or mute the thread https://github.com/notifications/unsubscribe-auth/AHAcYAL07v0ZVoOlfoLearBMmPY2wzRaks5u1QZtgaJpZM4Y-XG1 .

[SongFGH]

Thanks!

[Tomposon]

Do you understand norm now? @SongFGH

[SongFGH]

actually, it means we should give more attention to non-zero element in adjacent matrix

[Tomposon]

Oh, thank you very much!

[Tomposon]

But I find it use norm to multiply the final loss, not only to enlarge the non-zeros element loss like in SDNE. Did you exactly find the answer from the paper the author recommend to you?

self.cost = norm * tf.reduce_mean(tf.nn.weighted_cross_entropy_with_logits(logits=preds_sub, targets=labels_sub, pos_weight=pos_weight)) @SongFGH

[SongFGH]

I don't find the answer from the paper recommended. All I said is my guess! Supposeweight of zeros element loss is 1, weight of non-zeros element is norm .

[YH-UtMSB]

@SongFGH @Tomposon Well, there are (n_nodes * n_nodes) inter-node relationships in total, some of them are edges while others are not. Suppose we have RE edges and RN non-edges, such that

1. RE + RN = (n_nodes * n_nodes),
2. RN / RE = K.

The intuition behind norm, I guess, is to augment edges by replicating edges and corresponding predictions by K times, so that edges and the non-edge counterparts are balanced in terms of quantity. Of course, you don't have to manually do that since the argument pos_weight in tf.nn.weighted_cross_entropy_with_logits has attained an equivalent result.

Now we not only balanced training data but also incremented the size from (n_nodes * n_nodes) to (K*RE + RN) = (2 * RN). However, when applying tf.reduce_mean to the binary cross-entropy, the latter is only divided by (n_nodes * n_nodes), that's why we need to renormalize it by multiplying a term

norm = "size of original training set" / "size of augmented training set" = (n_nodes * n_nodes) / (2 * RN)

[udayshankars]

@hylBen Thank you so much for the explanation!