the conflict of protobuf version between apache orc and tensorflow

[j2cms]

when use tensorflow 1.15.5, it depend protobuf, the default version is 3.19.6 but the apache orc use protobuf 3.5.1 ( it seems too old)

they meet conflict as below:

Python(3937,0x7ff85c27bfc0) malloc: *** error for object 0x600002e394d8: pointer being freed was not allocated Python(3937,0x7ff85c27bfc0) malloc: *** set a breakpoint in malloc_error_break to debug

I found the similarity issues [https://github.com/apache/orc/issues/1425]

can we upgrade PROTOBUF version to 3.19.6?

[dongjoon-hyun]

Hi, @j2cms .

Which Apache ORC versions are you using now? IIUC, Apache ORC v2.x uses more higher versions of protobuf.

• Apache ORC 1.9.5 https://github.com/apache/orc/blob/c6904de5310ca8e6bcf77ca6fe2fc2335624e774/java/pom.xml#L135

• Apache ORC 2.0.0 https://github.com/apache/orc/blob/46eb6ff46c25a3c8763f8661dc5525bbe6a26550/java/pom.xml#L81

• Apache ORC 2.1.0 https://github.com/apache/orc/blob/2cb13946b71140be08b54111ff36fc17da5f09af/java/pom.xml#L82

[dongjoon-hyun]

BTW, FYI, Apache ORC community doesn't maintain PyPi packages.

[j2cms]

hi, @dongjoon-hyun , thank you for the reply.

I use the latest version of Apache ORC, I think the reason is PROTOBUF_VERSION in cmake_modules is 3.5.1

https://github.com/apache/orc/blob/b9641c6218a6ccb752822d18558e8cb8be4c8eec/cmake_modules/ThirdpartyToolchain.cmake#L27

can you upgrade it to 3.19.6 or later ?

[dongjoon-hyun]

Thank you, @j2cms .

I searched the community activity. It seems that it was tried here.

• https://github.com/apache/orc/pull/1857

[dongjoon-hyun]

It seems that we deprecated HDFS support to upgrade C++ protobuf at ORC 2.0.1 but we couldn't delete it in Apache Orc 2.x because we follow the semantic versioning.

• https://github.com/apache/orc/pull/1885

In this case, we need to wait until 3.0.0 because 2.0 and 2.1 is already out with BUILD_LIBHDFSPP.

cc @wgtmac

[dongjoon-hyun]

I filed the following.

• https://github.com/apache/orc/issues/2136

[wgtmac]

@j2cms If you have an installed protobuf somewhere, perhaps you may want to set one of these environment variables before building to avoid the default one.

[j2cms]

hi, @dongjoon-hyun , thank you for the reply.

I use the latest version of Apache ORC, I think the reason is PROTOBUF_VERSION in cmake_modules is 3.5.1

orc/cmake_modules/ThirdpartyToolchain.cmake

Line 27 in b9641c6

set(PROTOBUF_VERSION "3.5.1") can you upgrade it to 3.19.6 or later ?

I changed it to 3.19.6 as above and compile orc locally (on my MacOS), success. then, I compile pyorc use the orc I compiled,

finally It works. no protobuf version conflict with tensorflow now

maybe we should upgrade it to 3.19.6, not 3.21.12 in https://github.com/apache/orc/pull/1857

[dongjoon-hyun]

We started to discuss Apache ORC 2.2 release.

After releasing 2.2, we can start to re-discuss

• #1857
• #2136

in the scope of Apache ORC 3.0.