Error - bazel run

[karlie38]

Python 3.9.7 OS: Window

Having an error when running bazel to read RIEGELI files.

I followed the readme (install google sdk -> download data using 'gsutil' -> git clone code_contests -> bazel run -c opt :print_names_and_sources C:\tmp\dm-code_contests\dm-code_contests/code_contests_valid.riegeli ). I saw same issue with me (https://github.com/deepmind/code_contests/issues/4) and saw the answer of suggesting change of python version (3.x -> 2.x). But, "local variable 'python_include' is referenced before assignment" error also happened and I thought this problem and version issue have to be fixed.

Is there any way to read RIEGELI files easily?

[PhungVanDuy]

I just resolved this problem, you should make sure your gcc is a recent new version (my system with gcc-7.5 on Ubuntu 16.04).

In terms of python2 you also should install like this:

1. Remove bazel and reinstall

bazel clean --expunge
rm -rf ~/.cache/bazel

You can re-install follow this instruction

2. Install python2 dependency

sudo apt update && sudo apt install python-dev

I hope this can help you! Good luck!

[davidhchoi]

Do the above instructions work for you? In particular, you need to install both Python 3 and Python 2, and this is a not a problem that can be fixed in the repo itself.

[PhungVanDuy]

@davidhchoi I get this when I try to run execution/evalutation code, it's seems missing this file contest_problem.pb.h

Can you check this one?

[karlie38]

My python version is 3.x and my gcc version is latest. I also tested it in another environment (Ubuntu. 18.04 & Python version 3.x ). It worked when I followed the guide. However, in Window environment, it didn't work.

[davidhchoi]

@karlie38 what is your Python 2 version? Can you check that Python 2 is installed as well as Python 3?

@PhungVanDuy what is your bazel version, exact command line, and local changes (if any)? That is a proto file that should be automatically compiled into a header file by bazel.

[PhungVanDuy]

@davidhchoi My bazel version is bazel-5.0.0-linux-x86_64, can you check this one https://www.kaggle.com/code/duyphung/betacode-build ?

@davidhchoi I saw that many people facing with compiler problem, Can deepmind team release a docker version for everyone? I think it's will help a lot.

[davidhchoi]

For your first question, is it possible to try this directly on a Linux machine? Or does this happen outside of a notebook? It might be possible that creating the header for the proto fails for some reason (e.g. the build directory is not writeable). I assume load_data_test.cc also fails.

For the second, it's possible, but we might wait to learn a little bit more what issues people run into. I think specifying bazel/python 2/python 3/clang versions covers all issues currently (plus that we can't currently support non-Linux systems).

[PhungVanDuy]

@davidhchoi I have success to build execution code, I have a question that you have any plan to release code for CPP solution testing? As I known that currently this only support Python Solution right?

[davidhchoi]

The C++ sandbox is significantly more complicated, and more coupled with internal tools, so while we may release it in the future we currently can't promise anything. I think any kind of sandbox will work though; the main purpose of releasing solution testing is to demonstrate how the output checking works in case it isn't clear from the paper, rather than releasing all the sandboxes we use.

That said, I think we'd welcome pull requests to add a C++ sandbox (and also giving instructions for colab, other OSs, a docker image, etc. as asked in your last question)

[PhungVanDuy]

@davidhchoi Thank you for your information, I just need to ask because if we have a complete sandbox for the test maybe it's can help with some ideas to use RL to guide the decoder with execution score as a part of the reward. Maybe I can make docker image for everyone.

[davidhchoi]

RL to guide the encoder is an interesting idea, but I think Python evaluation should be enough to get you quite far on this so you can see if it's viable before investing in C++.

[PhungVanDuy]

Hi @davidhchoi ,

I am working on few ideas to improve AlphaCode so I have few research question that I want to discuss with you, if you don't mind can you give me your contact? It would be my pleasure to speak to you directly via email/discord or slack. Looking for your response.

Thanks.

[davidhchoi]

Sure. My contact information is in the AlphaCode paper. You can reach me at davidhchoi at deepmind.com.

[PhungVanDuy]

@davidhchoi thank you for your information, I have sent you an email via [email protected]. Looking for your response.

[DanialZohairen]

Do the above instructions work for you? In particular, you need to install both Python 3 and Python 2, and this is a not a problem that can be fixed in the repo itself.

I already have installed both python2 and python3 and I'm still getting the same error. I'm working on a windows10

[karlie38]

@DanialZohairen It didn't work out for me in window environment. I did it in Ubuntu. 18.04 & Python version 3.x ).

[davidhchoi]

Yes, as mentioned in https://github.com/deepmind/code_contests#supported-platforms we have only tested this on Linux, and aren't able to test this on other platforms like Windows. If someone manages to get it working on Windows though, we'd be happy to update the documentation with instructions.

[LaudemPax]

@karlie38 may I know which versions of gcc and bazel you used? I'm trying to create a docker image for this but I keep getting this error:

[karlie38]

My system has updated and i'm not sure about gcc version, but bazel version was bazel-5.0.0 and gcc version was 9.3.0. @LaudemPax

[LaudemPax]

In case anyone else needs it, I made a docker container as a workaround to get the bazel build working on Windows. Its on Docker Hub and has been used to train a model with the code_contests dataset on windows.

[wvaughn409]

In case anyone else needs it, I made a docker container as a workaround to get the bazel build working on Windows. Its on Docker Hub and has been used to train a model with the code_contests dataset on windows.

**YET ANOTHER UPDATE: I decided to just spin up an ubuntu vm and try this the right way, and I'm encountering the same error as with the Docker container, so I have created a new issue: https://github.com/deepmind/code_contests/issues/17

**NEW UPDATE: next problem I'm getting is that when I attempt to perform the execution/evaluation sample code like so I get an error:

docker run -it -v ${PWD}:/build -w /build laudempax/deepmind-bazel:ubuntu-20.04 run -c opt execution:solve_example /build/tmp/dm-code_contests/code_contests_valid.riegeli

Target //execution:solve_example up-to-date: bazel-bin/execution/solve_example INFO: Elapsed time: 92.776s, Critical Path: 23.83s INFO: 1001 processes: 320 internal, 681 processwrapper-sandbox. INFO: Build completed successfully, 1001 total actions INFO: Build completed successfully, 1001 total actions Failed: Gregor and Cryptography problem not found. Did you pass the validation dataset?

---

**UPDATE: I was able to get this working by running it this way in elevated Powershell (from the folder with the WORKSPACE file):

docker run -it -v ${PWD}:/build -w /build laudempax/deepmind-bazel:ubuntu-20.04 run -c opt :print_names_and_sources /build/tmp/dm-code_contests/code_contests_valid.riegeli

---

I tried out your Docker container, thank you by the way for doing that, but the command cannot seem to find the local .riegeli file I'm referencing: FileNotFoundError: [Errno 2] No such file or directory: 'C:\code_contests-main/tmp/dm-code_contests/code_contests_valid.riegeli'

Here is the modified command I executed from terminal: docker run -it -v %cd%:/build -w /build laudempax/deepmind-bazel:ubuntu-20.04 run -c opt :print_names_and_sources %cd%/tmp/dm-code_contests/code_contests_valid.riegeli

C:\code_contests-main>docker run -it -v %cd%:/build -w /build laudempax/deepmind-bazel:ubuntu-20.04 run -c opt :print_names_and_sources %cd%/tmp/dm-code_contests/code_contests_valid.riegeli Extracting Bazel installation... Starting local Bazel server and connecting to it... DEBUG: Rule 'com_github_grpc_grpc' indicated that a canonical reproducible form can be obtained by modifying arguments sha256 = "987ea637191b99f61ef24c54315c6eb7457eac11107513da1562bd2213163bde" DEBUG: Repository com_github_grpc_grpc instantiated at: /build/WORKSPACE:5:13: in <toplevel> Repository rule http_archive defined at: /root/.cache/bazel/_bazel_root/7b7747ec045ae606eb720a1222f56098/external/bazel_tools/tools/build_defs/repo/http.bzl:353:31: in <toplevel> INFO: Analyzed target //:print_names_and_sources (86 packages loaded, 2023 targets configured). INFO: Found 1 target... INFO: From ProtoCompile external/com_google_protobuf/python/google/protobuf/any_pb2.py: external/com_google_protobuf/python: warning: directory does not exist. INFO: From ProtoCompile external/com_google_protobuf/python/google/protobuf/descriptor_pb2.py: external/com_google_protobuf/python: warning: directory does not exist. INFO: From ProtoCompile external/com_google_protobuf/python/google/protobuf/timestamp_pb2.py: external/com_google_protobuf/python: warning: directory does not exist. INFO: From ProtoCompile external/com_google_protobuf/python/google/protobuf/wrappers_pb2.py: external/com_google_protobuf/python: warning: directory does not exist. INFO: From ProtoCompile external/com_google_protobuf/python/google/protobuf/struct_pb2.py: external/com_google_protobuf/python: warning: directory does not exist. INFO: From ProtoCompile external/com_google_protobuf/python/google/protobuf/source_context_pb2.py: external/com_google_protobuf/python: warning: directory does not exist. INFO: From ProtoCompile external/com_google_protobuf/python/google/protobuf/duration_pb2.py: external/com_google_protobuf/python: warning: directory does not exist. INFO: From ProtoCompile external/com_google_protobuf/python/google/protobuf/field_mask_pb2.py: external/com_google_protobuf/python: warning: directory does not exist. INFO: From ProtoCompile external/com_google_protobuf/python/google/protobuf/compiler/plugin_pb2.py: external/com_google_protobuf/python: warning: directory does not exist. INFO: From ProtoCompile external/com_google_protobuf/python/google/protobuf/type_pb2.py: external/com_google_protobuf/python: warning: directory does not exist. INFO: From ProtoCompile external/com_google_protobuf/python/google/protobuf/api_pb2.py: external/com_google_protobuf/python: warning: directory does not exist. INFO: From ProtoCompile external/com_google_protobuf/python/google/protobuf/empty_pb2.py: external/com_google_protobuf/python: warning: directory does not exist. INFO: From ProtoCompile external/com_google_riegeli/python/riegeli/records/records_metadata_pb2.py: external/com_google_protobuf/python: warning: directory does not exist. INFO: From Compiling riegeli/records/record_reader.cc: In file included from external/com_google_absl/absl/container/inlined_vector.h:54, from external/com_google_absl/absl/strings/cord.h:78, from external/com_google_riegeli/riegeli/records/record_reader.h:27, from external/com_google_riegeli/riegeli/records/record_reader.cc:15: external/com_google_absl/absl/container/internal/inlined_vector.h: In constructor 'riegeli::RecordReaderBase::RecordReaderBase()': external/com_google_absl/absl/container/internal/inlined_vector.h:433:5: warning: '<anonymous>.absl::lts_20211102::inlined_vector_internal::Storage<int, 1, std::allocator<int> >::data_' is used uninitialized in this function [-Wuninitialized] 433 | data_ = other_storage.data_; | ^~~~~ external/com_google_absl/absl/container/internal/inlined_vector.h: In constructor 'riegeli::RecordReaderBase::RecordReaderBase(riegeli::Closed)': external/com_google_absl/absl/container/internal/inlined_vector.h:433:5: warning: '<anonymous>.absl::lts_20211102::inlined_vector_internal::Storage<int, 1, std::allocator<int> >::data_' is used uninitialized in this function [-Wuninitialized] 433 | data_ = other_storage.data_; | ^~~~~ external/com_google_absl/absl/container/internal/inlined_vector.h: In member function 'void riegeli::RecordReaderBase::Reset(riegeli::Closed)': external/com_google_absl/absl/container/internal/inlined_vector.h:433:5: warning: '<anonymous>.absl::lts_20211102::inlined_vector_internal::Storage<int, 1, std::allocator<int> >::data_' is used uninitialized in this function [-Wuninitialized] 433 | data_ = other_storage.data_; | ^~~~~ Target //:print_names_and_sources up-to-date: bazel-bin/print_names_and_sources INFO: Elapsed time: 149.233s, Critical Path: 30.23s INFO: 716 processes: 21 internal, 695 processwrapper-sandbox. INFO: Build completed successfully, 716 total actions INFO: Build completed successfully, 716 total actions Traceback (most recent call last): File "/root/.cache/bazel/_bazel_root/7b7747ec045ae606eb720a1222f56098/execroot/__main__/bazel-out/k8-opt/bin/print_names_and_sources.runfiles/__main__/print_names_and_sources.py", line 46, in <module> _print_names_and_sources(sys.argv[1:]) File "/root/.cache/bazel/_bazel_root/7b7747ec045ae606eb720a1222f56098/execroot/__main__/bazel-out/k8-opt/bin/print_names_and_sources.runfiles/__main__/print_names_and_sources.py", line 39, in _print_names_and_sources for problem in _all_problems(filenames): File "/root/.cache/bazel/_bazel_root/7b7747ec045ae606eb720a1222f56098/execroot/__main__/bazel-out/k8-opt/bin/print_names_and_sources.runfiles/__main__/print_names_and_sources.py", line 32, in _all_problems reader = riegeli.RecordReader(io.FileIO(filename, mode='rb'),) FileNotFoundError: [Errno 2] No such file or directory: 'C:\\code_contests-main/tmp/dm-code_contests/code_contests_valid.riegeli'

[Amandaynzhou]

The C++ sandbox is significantly more complicated, and more coupled with internal tools, so while we may release it in the future we currently can't promise anything. I think any kind of sandbox will work though; the main purpose of releasing solution testing is to demonstrate how the output checking works in case it isn't clear from the paper, rather than releasing all the sandboxes we use.

That said, I think we'd welcome pull requests to add a C++ sandbox (and also giving instructions for colab, other OSs, a docker image, etc. as asked in your last question)

Hi! Is there any plan to release the c++ sandbox currently?