how to decide whether an operator is trainable

[breadbread1984]

I am confusing about whether I shall put an operator into trainable_ops or non_trainable_ops. How do you decide it? I saw Scale operator was put in non_trainable_ops. Shall I put Pow in non_trainable_ops as well?

[leovandriel]

That's a good question. This trainable/non_trainable thing is a bit of a hack. It's mainly used to decide for which operators we need a to add a gradient in AddGradientOps. In general the rule is: if it's an operator that is a core part of the model output, then it's probably in trainable. I don't recall why Scale is not in trainable. Pow should be in trainable.