Feature request: optimizer support with slot variables in different dtype to the variable.

[georgepaw]

Currently an assumption is made that slot variables have the same dtype as the variable, which prevents experimentation.

From digging around a solution is to:

1. Extend SlotVariableReference proto in tensorflow/core/protobuf/trackable_object_graph.proto to include the dtype and set that field when saving/loading trackable objects in TF
2. Extend the add_slot function to take dtype=None as a kwarg and then set the dtype of the created variable to dtype or var.dtype to preserve backwards compatibility.
3. Set the dtype arg when calling add_slot in tensorflow/python/saved_model/load.py

More than happy to implement this, especially if you guys are happy with the proposed solution here.

[tilakrayal]

@georgepaw , Can you please share a Use-Case that supports your statement so that the issue can be easily understood? Thanks!

[georgepaw]

One could write an optimizer (for example Adam) for a model which has the weights and gradients in fp16, but the slot variables might have to be in higher precision - for example fp32

[qlzh727]

Add Chen for more inputs.

[georgepaw]

Any thoughts on the proposed solution?

[chenmoneygithub]

Sorry for missing this issue (somehow it slipped away).

I like the idea of flexible dtype, actually in the experimental Keras optimizer, there is a method add_variable that suits your needs, could you check if that works?

Thanks, and sorry again!