tensorboard icon indicating copy to clipboard operation
tensorboard copied to clipboard

protoc compiler is too old for tensorflow 2.12

Open jzhoulon opened this issue 1 year ago • 6 comments

tensorflow 2.12 has upgrade its protoc compiler to 3.21.9 while tensorboard still keeps 3.19.6, this will make profiler data(xplane.xxx.pb) failed to load by tensorboard.
tensorboard proto: https://github.com/tensorflow/tensorboard/blob/56b38b75dfb993d0da6a241114bb475d072b5ab9/WORKSPACE#L138 tensorflow proto https://github.com/tensorflow/tensorflow/blob/e8473900214fb5508699ebeafd7ea5f3ef0c2fd1/tensorflow/workspace2.bzl#L463

The error messeage is as follows:

ackages/tensorboard_plugin_profile/protobuf/diagnostics_pb2.py", line 36, in <module>
    _descriptor.FieldDescriptor(
  File "/home/subratag/mc3/envs/23ww14/lib/python3.9/site-packages/google/protobuf/descriptor.py", line 561, in __new__
    _message.Message._CheckCalledFromGeneratedFile()
TypeError: Descriptors cannot not be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
 1. Downgrade the protobuf package to 3.20.x or lower.
 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower). 

jzhoulon avatar Apr 12 '23 01:04 jzhoulon

I think the title should be 2.12 not 1.12, correct? I'll edit if that was a typo.

We've seen this before, and its due to the profiler plugin using an old protobuf compiler. TB can not update the protobuf compiler since we need to maintain backward compatibility with older version of TF. The profiler plugin was updated to a working version last week, can you check if you have that? Specifically you want to see: https://github.com/tensorflow/profiler/blob/master/plugin/setup.py#L26

ericdnielsen avatar Apr 12 '23 14:04 ericdnielsen

I think the title should be 2.12 not 1.12, correct? I'll edit if that was a typo.

We've seen this before, and its due to the profiler plugin using an old protobuf compiler. TB can not update the protobuf compiler since we need to maintain backward compatibility with older version of TF. The profiler plugin was updated to a working version last week, can you check if you have that? Specifically you want to see: https://github.com/tensorflow/profiler/blob/master/plugin/setup.py#L26

sorry for the typo, it is 2.12, i edit it. thanks

jzhoulon avatar Apr 14 '23 02:04 jzhoulon

@ericdnielsen I install the profiler plugin and pip install the protobuf==3.20.1, the protoc issue is disappeared, however the profiler data can't show in tensorboard, I have host.xplane.pb in log dir, however, it reports no event files

======================================================================
Processing event files... (this can take a few minutes)
======================================================================

No event files found within logdir data

jzhoulon avatar Apr 20 '23 02:04 jzhoulon

At this point I think we should hand you off to the profiler team: https://github.com/tensorflow/profiler

ericdnielsen avatar Apr 20 '23 14:04 ericdnielsen

Hi, any plans to upgrade protobuf to >=3.20? It was release more than 1 year ago and some other libraries (eg. streamlit) require at least 3.20 thanks

dpinol avatar Jul 06 '23 09:07 dpinol

Bumping this as well, we use protobuf 4.21.3 and we keep running into the same issue. Are there plans of bumping the protoc compiler for TB?

ahmedetefy avatar Jul 25 '23 14:07 ahmedetefy