What makes TensorVariables `iterable`

[michaelosthege]

Describe the issue:

These are as expected:

from collections,abc import Iterable
>>> isinstance(pt.constant([1,2,3]), Iterable)
True
>>> isinstance(pytensor.shared(np.array([1,2,3])), Iterable)
True

But a purely symbolic at.vector() that doesn't have a value yet raises and error when attempting to iterate it, because it has no length.

And even worse, a pt.scalar() should never be iterable to begin with.

Reproducable code example:

>>> isinstance(pt.vector(), Iterable)
True
>>> isinstance(pt.scalar(), Iterable)
True

Error message:

No response

PyTensor version information:

main

Context for the issue:

No response

[ricardoV94]

Vectors are sometimes iterable if get_vector_length works.

import pytensor.tensor as pt

x = pt.zeros(5)
for i in x:
    print(i.eval())

Or

x = pt.vector(shape=(5,))
for i in x:
    print(i.eval({x: [0, 1, 2, 3, 4]}))

[ferrine]

Being iterable works when shapes are unpacked but I think shapes should be special vector classes

[ricardoV94]

Being iterable works when shapes (the leading dimension) are static. What do you mean unpacked?

[ricardoV94]

I would suggest closing this issue in favor of one that discusses tuple types for shapes

[ricardoV94]

@michaelosthege is there some actionable action you want here?

[michaelosthege]

Not sure if there is. Maybe this is a case of "it would take ∞ hours of refactoring to fix"

The pt.vector() and pt.scalar() are of type TensorVariable, and count as issubclass(TensorVariable, Iterable), because pytensor.tensor.variable._tensor_py_operators implements a __iter__ method.

From a formal type system perspective one could distinguish tensors with symbolic vs. static shape.

---

I'm a bit rusty on the PyTensor types, but let me ask these two questions:

• Should a pt.scalar() create a tensor variable to begin with?
• Where do we draw the line between Variable and TensorVariable?

There's a ton of code, particularly in PyMC that expects things to be TensorVariable, but often types we only annotate Variable. We have quite a mess on these things, and maybe it's because we don't have a shared understanding of when to use/expect one or the other.

[ricardoV94]

pt.scalar creates the equivalent to a 0d numpy array, it's totally well defined. There is pytensor.scalar.float64 (or any other dtype) that would create the equivalent to what you may be thinking as scalars.

Variable is the general type for any object that is either an input or output to an Apply Node. Variables have different types, ScalarType, TensorType, SliceType, NoneTypeT, SparseTensorType, and so on... For some cases we are doing something funky with Python inheritance so that a Variable with a TensorType is a TensorVariable. For other types like SliceType we are not doing that.

I don't see a good reason not to do for all. But that's probably already beyond your question?

[michaelosthege]

Okay, so right now we're mirroring NumPy in that a NDarray may be a 0-dimensional (scalar) thing.

Just formally, I would expect a tensor to be at least 1-dimensional, making it length-measurable, indexable, iterable and so on.

I think it's okay that a vector is only iterable if it's length isn't symbolic, but type(at.scalar()) shouldn't even have these methods. We could define them in a different class than _tensor_py_operators and refactor such that type(pt.vector()) != type(pt.scalar()).

Would this translate to something actionable?

[ricardoV94]

A 0d tensor is just fine, it's an array object that holds a single item. You can even update it with x[()] = 5.0 which you obviously can't with float/integers that are atomic/immutable objects.

Regarding iteration, we should be able to iterate over anything that we can index along the first dimension (so actually matrices, higher-dimensional tensors are also fine). Historically vectors were treated as special cases because we use them to represent shapes, and we often need to know how many entries there are in a shape for stuff like x.reshape(y.shape). Theano didn't have static length than 1 vs None so they had a lot of hacky logic to get around this limitatation, which is where get_vector_length comes up.

0d arrays should'n be iterable. They are not in Numpy nor PyTensor already:

import numpy as np

x = np.array(0.5)  # TypeError: iteration over a 0-d array
for y in x:
    print(y)   

import pytensor.tensor as pt

x = pt.scalar("x")
for i in x:  # TypeError: TensorType does not support iteration.
    print(i)        

Message above could be more precise > For Scalars: Scalar TensorType (ndim=0) does not support iteration For everything else: TensorType without static shape on the leading dimension does not support iteration: {tensortype}.

[ricardoV94]

So in practice a variable these days is iterable if get_vector_length succeeds, and that does not depend exclusively on the type, because it is allowed to go up the graph and figure out the length from the operations that it finds.

We just know that scalars are never iterable, nor are >1d tensors, although the last one should be allowed.