data icon indicating copy to clipboard operation
data copied to clipboard

Enforce the version of Protobuf via the optional dependency

Open jayagami opened this issue 2 years ago • 3 comments

🐛 Describe the bug

TypeError: Descriptors cannot not be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
 1. Downgrade the protobuf package to 3.20.x or lower.
 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).

https://github.com/pytorch/data/blob/d9bbbecf64d0149795dc65ba390b50bc9e176e95/torchdata/datapipes/iter/util/protobuf_template/_tfrecord_example_pb2.py#L39

Versions

Lastest version.

jayagami avatar Apr 20 '23 17:04 jayagami

IIRC, the last time I checked the Tensorflow still requires protobuf version < 3.20

ejguan avatar Apr 20 '23 17:04 ejguan

I think you are right, but it might be better to add version control and restrict the protobuf version to 3.20.

by the way can I get the number of tfrecord?

jayagami avatar Apr 20 '23 17:04 jayagami

It makes sense to add optional dependency specification to TorchData. https://setuptools.pypa.io/en/latest/userguide/dependency_management.html#optional-dependencies

ejguan avatar Apr 20 '23 18:04 ejguan