Discussion: immutable/mutable tensors

[dsyme]

Currently tensors are immutable, though may share the same underlying data.

We do have some localized mutation through .reverseDiff() that creates a new tensor with a mutable register chain to receive adjoints from a later matching .reverse(). However I'll ignore that as it's "nice" mutation, because it is localised in the sense that (1) if you don't use these constructs, everything is immutable, and (2) if you do use them according to the rules then they won't interfere.

Now, I'm sure that at some point we will need to introduce localized mutation on tensors. Indeed we already use localized mutation internally in Tensor.fs, but again it is local, not global.

I propose we do it somthing like this?

1. tensors are by default immutable, and we check for this and raise exceptions if you try to mutate
2. if you do tensor.mutableLike() then you get a mutable tensor - still type Tensor but mutation operations no longer throw exceptions. A isMutable flag can also be passed into zerosLike and friends to get a tensor that starts off mutable.
3. a tensor.unsafeImmutable() gets you back to an immutable tensor, sharing the same underlying data

Like reverse this API is safe if you play by the rules.

I also propose that we eventually somehow keep a flag to report when we know that a tensor has a zero value, basically tensor.isKnownZero, likewise tensor.isKnownOne. I expect these will be just too useful for peep-hole optimization purposes not to propogate them, and they can help us simplify how we write extensions (see this. But we can only trust these flags if tensors are immutable....

[gbaydin]

Just to check: the immutability mentioned here also relates to in-place operations, is that correct?

[dsyme]

Just to check: the immutability mentioned here also relates to in-place operations, is that correct?

Yes, in place operations = mutable. To use an in-place operation you'd need to set a tensor up to be mutable (which would not be the default). Internally in the implementation we can likely cheat (though I don't think we'd have to - it's pretty obvious where mutation on a tensor is desired).

We'd need a flag on each tensor value, either stored in the backend or the primal.