samjabrahams
During TF build with bazel, protoc seems not built fresh ?
Hi,
Thank you for comprehensive instructions. I am newbie on this :) and I may be doing something wrong but I could not yet compile TF due to the issue below.
Thank you in advance for any suggestions !
Cheers Ersin
Describe the Issue
When following step-by-step instructions, I managed to build bazel. However, during TF build step at some point I get following error (copy/pasted from ~/.cache/bazel/_bazel_pi/369d1e70a26639ff91587139b0224d09/command.log)
=======
ERROR: /home/pi/.cache/bazel/_bazel_pi/369d1e70a26639ff91587139b0224d09/external/io_bazel_rules_closure/java/io/bazel/rules/closure/webfiles/s
erver/BUILD:48:1: error executing shell command: 'JAR='external/local_jdk/bin/jar' OUTPUT='bazel-out/host/bin/external/io_bazel_rules_closure/java/io/bazel/r
ules/closure/webfiles/server/libbuild_info_java_proto_srcjar.srcjar' PROTO_COMPILER='exter...' failed: bash failed: error executing command
(cd /home/pi/.cache/bazel/_bazel_pi/369d1e70a26639ff91587139b0224d09/execroot/tensorflow &&
exec env -
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/games:/usr/games
/bin/bash -c 'JAR='''external/local_jdk/bin/jar''' OUTPUT='''bazel-out/host/bin/external/io_bazel_rules_closure/java/io/bazel/rules/closure/webfiles/ser
ver/libbuild_info_java_proto_srcjar.srcjar''' PROTO_COMPILER='''external/com_google_protobuf_protoc/bin/protoc''' SOURCE='''external/io_bazel_rules_closu
re/java/io/bazel/rules/closure/webfiles/server/build_info.proto''' INCLUDES='''-I. -Iexternal/io_bazel_rules_closure''' bazel-out/host/bin/external/io_baz
el_rules_closure/closure/private/gensrcjar'): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1.
external/com_google_protobuf_protoc/bin/protoc: 1: external/com_google_protobuf_protoc/bin/protoc: cannot create �/@@��: Directory nonexistent
external/com_google_protobuf_protoc/bin/protoc: 1: external/com_google_protobuf_protoc/bin/protoc: ELF: not found
external/com_google_protobuf_protoc/bin/protoc: 2: external/com_google_protobuf_protoc/bin/protoc: Syntax error: ")" unexpected
gensrcjar: proto_compiler failed
Target //tensorflow/tools/pip_package:build_pip_package failed to build
=======
After noticing strange characters, I checked protoc binary (at ~/.cache/bazel/_bazel_pi/369d1e70a26639ff91587139b0224d09/external/com_google_protobuf_protoc/bin) and found that it somehow has been copied from somewhere rather than getting compiled from its source :
-rwxr-x--- 1 pi pi 4114152 Sep 25 2016 protoc
and for wrong architecture (64bit and not for arm):
file protoc protoc: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.24, BuildID[sha1]=afe1ae3a88d6b51b14fb21830cc133d9107e6ce6, stripped
I am guessing that is the reason. Maybe in some config file an adjustment is needed to tell bazel to compile protoc rather than download its executable form.
Steps to Reproduce
At step-by-step instructions, issue occurs deterministically at this step:
bazel build -c opt --copt="-mfpu=neon-vfpv4" --copt="-funsafe-math-optimizations" --copt="-ftree-vectorize" --copt="-fomit-frame-pointer" --local_resources 1024,1.0,1.0 --verbose_failures tensorflow/tools/pip_package:build_pip_package
Hardware/Software Info
Please provide the following information about your Raspberry Pi setup:
- Raspberry Pi model: 3
- Operating System used: Raspbian GNU/Linux 8 (jessie)
- Version of Python used: 3.4
- SD card memory size: 32
- Size of USB/other device used as swap (if building from source): 16
- TensorFlow git commit hash (if building from source): Cloned from master on 6/18/2017
Relevant Console Output/Logs
Thanks for opening this up @beyret - unfortunately, I'm not in a position to be able to walk through the steps on a Pi of my own for the next few weeks, but I'll try to come up with some suggestions that might help.
The first is to try using a different version of Bazel- If you used 0.4.5, try 0.5.1, the newest version. If you did 0.5.1, try the reverse. If those don't work, let me know and we'll try more things from there.
Thank you @samjabrahams for quick follow-up :) Just an update after some more progress:
-
I compiled Bazel 0.5.1. Encountered same issue (the protoc executable is obtained from somewhere instead of being freshly built).
-
I went ahead and compiled protoc executable by doing this:
cd ~/.cache/bazel/_bazel_pi/369d1e70a26639ff91587139b0224d09/external/protobuf bazel build protoc
- And confirmed that it is built for ARM 32bit:
file bazel-bin/protoc
bazel-bin/protoc: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 2.6.32, BuildID[sha1]=5d9674ac8c36cc8637c6bb6dc37bb60d2c7ad7c7, not stripped
- Then copied it over to where bazel for TF uses it from (hoping that it won't overwrite)
cp bazel-bin/protoc ~/.cache/bazel/_bazel_pi/369d1e70a26639ff91587139b0224d09/external/com_google_protobuf_protoc/bin
- And invoked the TF build again
bazel build ... tensorflow/tools/pip_package:build_pip_package
- I think it went passed the error stage but later down the road it failed with seemingly another issue:
ERROR: /home/pi/.cache/bazel/_bazel_pi/369d1e70a26639ff91587139b0224d09/external/io_bazel_rules_closure/java/io/bazel/rules/closure/BUILD:62:1: Building external/io_bazel_rules_closure/java/io/bazel/rules/closure/libbuild_info_java_proto.jar (1 source jar) failed: Worker process sent response with exit code: 1. /io/bazel/rules/closure/BuildInfo.java:767: error: no suitable method found for parseFrom(ByteBuffer) return PARSER.parseFrom(data); ^ method Parser.parseFrom(CodedInputStream) is not applicable (argument mismatch; ByteBuffer cannot be converted to CodedInputStream) method Parser.parseFrom(ByteString) is not applicable (argument mismatch; ByteBuffer cannot be converted to ByteString) method Parser.parseFrom(byte[]) is not applicable (argument mismatch; ByteBuffer cannot be converted to byte[]) method Parser.parseFrom(InputStream) is not applicable (argument mismatch; ByteBuffer cannot be converted to InputStream) /io/bazel/rules/closure/BuildInfo.java:773: error: no suitable method found for parseFrom(ByteBuffer,ExtensionRegistryLite) return PARSER.parseFrom(data, extensionRegistry); ^ method Parser.parseFrom(CodedInputStream,ExtensionRegistryLite) is not applicable (argument mismatch; ByteBuffer cannot be converted to CodedInputStream) method Parser.parseFrom(ByteString,ExtensionRegistryLite) is not applicable (argument mismatch; ByteBuffer cannot be converted to ByteString) method Parser.parseFrom(byte[],ExtensionRegistryLite) is not applicable (argument mismatch; ByteBuffer cannot be converted to byte[]) method Parser.parseFrom(InputStream,ExtensionRegistryLite) is not applicable (argument mismatch; ByteBuffer cannot be converted to InputStream)
- Is there a particular stable branch/date of tensorflow codestream you would recommend ? Maybe me trying to use the master code stream is not a good idea. By nature, it must have a lot of flux and things may be yet to be discovered/settled.
Just an update: I've been able to compile TF :)
-
I've used r1.2 branch instead of master (by doing "git checkout r1.2" after "git clone" of tensorflow repo). I anyway do not need very latest version at this point.
-
At 1.2, there is apparently a known (and recently fixed) issue there w.r.t Eigen (issue #101 thx @SteveBetter !). I've applied the patch from its raw diff as such:
cd ~/.cache/bazel/_bazel_pi/369d1e70a26639ff91587139b0224d09/external/eigen_archive git apply <raw_diff_file>
- Then TF build completed :)
bazel build ... tensorflow/tools/pip_package:build_pip_package
- After creating the wheel per instructions, I used pip3 to install it:
sudo pip3 install /tmp/tensorflow_pkg/tensorflow-1.2.0-cp34-cp34m-linux_armv7l.whl