Use latest compatible grpcio-tools version and unlock support for Python 3.11
Currently blocked to 1.48.2 because of TF 2.x limitation of protobuf < 3.20 (and TF is currently used in Ray RL Lib). This also limit python version to 3.11 as it requires a newer grpcio version (above 1.48)
Files affected:
- Engine Docker File
- diambra-engine python package
build-pip-package - GH actions limited to python 3.10
Other relevant info
Precondition on protobuf changed between: grpcio-tools 1.48.2 https://github.com/grpc/grpc/blob/5e97a6003d0e44bdf3c4234d37b9509d02bf55ea/tools/distrib/python/grpcio_tools/setup.py#L293 and 1.49: https://github.com/grpc/grpc/blob/8f8edfd04b46ee67f90454b3f6a70aa58ff82c2d/tools/distrib/python/grpcio_tools/setup.py#L293
Compatibility problem of protobuf https://stackoverflow.com/questions/71759248/importerror-cannot-import-name-builder-from-google-protobuf-internal
To sum this up:
- ray-rllib needs tensorflow 2.x
- tensorflow 2.x needs
protobuf (<3.20,>=3.9.2)
$ for v in 2.11.0 2.10.1 2.10.0 2.9.3 2.9.2 2.9.1 2.8.4; do echo "$v: $(curl -s https://pypi.org/pypi/tensorflow/$v/json | jq .info.requires_dist|grep protobuf)"; done
2.11.0: "protobuf (<3.20,>=3.9.2)",
2.10.1: "protobuf (<3.20,>=3.9.2)",
2.10.0: "protobuf (<3.20,>=3.9.2)",
2.9.3: "protobuf (<3.20,>=3.9.2)",
2.9.2: "protobuf (<3.20,>=3.9.2)",
2.9.1: "protobuf (<3.20,>=3.9.2)",
2.8.4: "protobuf (<3.20,>=3.9.2)",
- Since there can be only one version installed, we need to depend on
protobuf (<3.20,>=3.9.2)as well - grpcio-tools generates the diambra-arena-engine code assuming a specific protobuf version
- grpcio-tools depends on that protobuf version:
$ for v in 1.51.1 1.50.0 1.49.1 1.48.2; do echo "$v: $(curl -s https://pypi.org/pypi/grpcio-tools/$v/json | jq .info.requires_dist|grep protobuf)"; done
1.51.1: "protobuf (<5.0dev,>=4.21.6)",
1.50.0: "protobuf (<5.0dev,>=4.21.6)",
1.49.1: "protobuf (<5.0dev,>=4.21.3)",
1.48.2: "protobuf (<4.0dev,>=3.12.0)",
Therefor we need to:
- Generate diambra-arena-engine using a grpcio-tools version that generate code for a version matching
protobuf (<3.20,>=3.9.2), e.g1.48.2 - Use that version's dependency (
protobuf (<4.0dev,>=3.12.0)in our setup.py
That will allow users to install ray-rllib where pip dep resolver should install protobuf <3.20,>=3.9.2 while allowing users not depending on ray/tf2 to also install <=3.12.0 and >=3.20.