arena icon indicating copy to clipboard operation
arena copied to clipboard

Use latest compatible grpcio-tools version and unlock support for Python 3.11

Open discordianfish opened this issue 2 years ago • 1 comments

Currently blocked to 1.48.2 because of TF 2.x limitation of protobuf < 3.20 (and TF is currently used in Ray RL Lib). This also limit python version to 3.11 as it requires a newer grpcio version (above 1.48)

Files affected:

  • Engine Docker File
  • diambra-engine python package build-pip-package
  • GH actions limited to python 3.10

Other relevant info

Precondition on protobuf changed between: grpcio-tools 1.48.2 https://github.com/grpc/grpc/blob/5e97a6003d0e44bdf3c4234d37b9509d02bf55ea/tools/distrib/python/grpcio_tools/setup.py#L293 and 1.49: https://github.com/grpc/grpc/blob/8f8edfd04b46ee67f90454b3f6a70aa58ff82c2d/tools/distrib/python/grpcio_tools/setup.py#L293

Compatibility problem of protobuf https://stackoverflow.com/questions/71759248/importerror-cannot-import-name-builder-from-google-protobuf-internal

discordianfish avatar Feb 06 '23 12:02 discordianfish

To sum this up:

  • ray-rllib needs tensorflow 2.x
  • tensorflow 2.x needs protobuf (<3.20,>=3.9.2)
$ for v in 2.11.0 2.10.1 2.10.0 2.9.3 2.9.2 2.9.1 2.8.4; do echo "$v: $(curl -s https://pypi.org/pypi/tensorflow/$v/json | jq .info.requires_dist|grep protobuf)"; done
2.11.0:   "protobuf (<3.20,>=3.9.2)",
2.10.1:   "protobuf (<3.20,>=3.9.2)",
2.10.0:   "protobuf (<3.20,>=3.9.2)",
2.9.3:   "protobuf (<3.20,>=3.9.2)",
2.9.2:   "protobuf (<3.20,>=3.9.2)",
2.9.1:   "protobuf (<3.20,>=3.9.2)",
2.8.4:   "protobuf (<3.20,>=3.9.2)",
  • Since there can be only one version installed, we need to depend on protobuf (<3.20,>=3.9.2) as well
  • grpcio-tools generates the diambra-arena-engine code assuming a specific protobuf version
  • grpcio-tools depends on that protobuf version:
$ for v in 1.51.1 1.50.0 1.49.1 1.48.2; do echo "$v: $(curl -s https://pypi.org/pypi/grpcio-tools/$v/json | jq .info.requires_dist|grep protobuf)"; done
1.51.1:   "protobuf (<5.0dev,>=4.21.6)",
1.50.0:   "protobuf (<5.0dev,>=4.21.6)",
1.49.1:   "protobuf (<5.0dev,>=4.21.3)",
1.48.2:   "protobuf (<4.0dev,>=3.12.0)",

Therefor we need to:

  1. Generate diambra-arena-engine using a grpcio-tools version that generate code for a version matching protobuf (<3.20,>=3.9.2), e.g 1.48.2
  2. Use that version's dependency (protobuf (<4.0dev,>=3.12.0) in our setup.py

That will allow users to install ray-rllib where pip dep resolver should install protobuf <3.20,>=3.9.2 while allowing users not depending on ray/tf2 to also install <=3.12.0 and >=3.20.

discordianfish avatar Feb 06 '23 15:02 discordianfish