Cxx.jl
Cxx.jl copied to clipboard
JuliaInterop
Cxx fails to build on fedora/linux
It looks like Cxx needs the julia source to build, but fedora linux packages don't provide it.
julia> Pkg.build("Cxx")
INFO: Building Cxx
writing path.jl file
Tuning for julia installation at /usr/bin with sources possibly at /usr/bin/../..
INFO: Building julia binary build
Not building debug library because corresponding julia DEBUG library does not exist.
Not building release library because corresponding julia RELEASE library does not exist.
To build, simply run the build again once the library at
To build, simply run the build again once the library at
/libjulia-debug.so
/libjulia.so
has been built.
has been built.
cc -E -P -I/usr/bin/../../src/support -I/usr/bin/../include -Isrc/clang-3.7.1/lib -Ibuild/clang-3.7.1/include -Isrc/clang-3.7.1/include -Isrc/llvm-3.7.1/include -Ibuild/llvm-3.7.1/include -DJULIA ../src/cenumvals.jl.h > build/clang_constants.jl
/bin/sh: build/clang_constants.jl: No such file or directory
BuildBootstrap.Makefile:175: recipe for target 'build/clang_constants.jl' failed
make: *** [build/clang_constants.jl] Error 1
=================================================[ ERROR: Cxx ]=================================================
LoadError: failed process: Process(`make -j4 -f BuildBootstrap.Makefile BASE_JULIA_BIN=/usr/bin BASE_JULIA_SRC=/usr/bin/../..`, ProcessExited(2)) [2]
while loading /home/nbecker/.julia/v0.5/Cxx/deps/build.jl, in expression starting on line 41
Running the command manually:
[nbecker@nbecker2 deps]$ make -j4 -f BuildBootstrap.Makefile BASE_JULIA_BIN=/usr/bin BASE_JULIA_SRC=/usr/bin/../..
BuildBootstrap.Makefile:6: /usr/bin/../../deps/Versions.make: No such file or directory
BuildBootstrap.Makefile:7: /usr/bin/../../Make.user: No such file or directory
make: *** No rule to make target '/usr/bin/../../Make.user'. Stop.
What can I do? Does Cxx really need the original build tree from Julia?
Turns out some limited changes are needed to make Cxx build with Julia from the RPM package. Here are the hacks I had to apply for that:
diff --git a/deps/BuildBootstrap.Makefile b/deps/BuildBootstrap.Makefile
index 8a43ed3..daf8a1a 100644
--- a/deps/BuildBootstrap.Makefile
+++ b/deps/BuildBootstrap.Makefile
@@ -40,7 +40,7 @@ endif
CXXJL_CPPFLAGS = -I$(JULIA_SRC)/src/support -I$(BASE_JULIA_BIN)/../include
ifeq ($(JULIA_BINARY_BUILD),1)
-LIBDIR := $(BASE_JULIA_BIN)/../lib/julia
+LIBDIR := $(BASE_JULIA_BIN)/../lib64/julia
else
LIBDIR := $(BASE_JULIA_BIN)/../lib
endif
@@ -92,7 +92,7 @@ else
CLANG_CMAKE_OPTS += -DLLVM_TABLEGEN_EXE=$(BASE_JULIA_BIN)/../tools/llvm-tblgen
endif
-JULIA_LDFLAGS = -L$(BASE_JULIA_BIN)/../lib -L$(BASE_JULIA_BIN)/../lib/julia
+JULIA_LDFLAGS = -L$(BASE_JULIA_BIN)/../lib64 -L$(BASE_JULIA_BIN)/../lib64/julia -L/usr/lib64/llvm3.7/
$(LLVM_CLANG_TAR): | src
curl -Lo $@ $(LLVM_SRC_URL)/$(notdir $@)
@@ -116,7 +116,7 @@ JULIA_LDFLAGS += -Lbuild/clang-$(LLVM_VER)/lib
CXXJL_CPPFLAGS += -Isrc/clang-$(LLVM_VER)/lib -Ibuild/clang-$(LLVM_VER)/include \
-Isrc/clang-$(LLVM_VER)/include
else # BUILD_LLVM_CLANG
-JULIA_LDFLAGS = -L$(BASE_JULIA_BIN)/../lib -L$(BASE_JULIA_BIN)/../lib/julia
+JULIA_LDFLAGS = -L$(BASE_JULIA_BIN)/../lib64 -L$(BASE_JULIA_BIN)/../lib64/julia
CXXJL_CPPFLAGS += -I$(JULIA_SRC)/deps/srccache/llvm-$(LLVM_VER)/tools/clang/lib \
-I$(JULIA_SRC)/deps/llvm-$(LLVM_VER)/tools/clang/lib
endif
@@ -147,7 +147,7 @@ LLVM_LIB_NAME := LLVM-$(CXX_LLVM_VER)
endif
LDFLAGS += -l$(LLVM_LIB_NAME)
-LIB_DEPENDENCY += $(LIBDIR)/lib$(LLVM_LIB_NAME).$(SHLIB_EXT)
+LIB_DEPENDENCY += /usr/lib64/llvm3.7/libLLVM-3.7.1.so
usr/lib:
@mkdir -p $(CURDIR)/usr/lib/
@@ -169,7 +169,7 @@ ifeq ($(BUILD_LLDB),1)
LINKED_LIBS += $(LLDB_LIBS)
endif
-ifneq (,$(wildcard $(BASE_JULIA_BIN)/../lib/libjulia.$(SHLIB_EXT)))
+ifneq (,$(wildcard $(BASE_JULIA_BIN)/../lib64/libjulia.$(SHLIB_EXT)))
usr/lib/libcxxffi.$(SHLIB_EXT): build/bootstrap.o $(LIB_DEPENDENCY) | usr/lib
@$(call PRINT_LINK, $(CXX) -shared -fPIC $(JULIA_LDFLAGS) -ljulia $(LDFLAGS) -o $@ $(WHOLE_ARCHIVE) $(LINKED_LIBS) $(NO_WHOLE_ARCHIVE) $< )
else
@@ -180,7 +180,7 @@ usr/lib/libcxxffi.$(SHLIB_EXT):
@echo "has been built."
endif
-ifneq (,$(wildcard $(BASE_JULIA_BIN)/../lib/libjulia-debug.$(SHLIB_EXT)))
+ifneq (,$(wildcard $(BASE_JULIA_BIN)/../lib64/libjulia-debug.$(SHLIB_EXT)))
usr/lib/libcxxffi-debug.$(SHLIB_EXT): build/bootstrap.o $(LIB_DEPENDENCY) | usr/lib
@$(call PRINT_LINK, $(CXX) -shared -fPIC $(JULIA_LDFLAGS) -ljulia-debug $(LDFLAGS) -o $@ $(WHOLE_ARCHIVE) $(LINKED_LIBS) $(NO_WHOLE_ARCHIVE) $< )
else
Of course this is not a general solution. Discussing with @Keno, we found out that LIB_DEPENDENCY should be determined via Libdl.dlpath("LLVM-$LLVM_VER") and that directory added to JULIA_LDFLAGS. ../lib should be replaced with Base.LIBDIR.
Finally, note that currently you also need to call mkdir ~/.julia/Cxx/deps/build manually.
Unfortunately, your patch does not seem to apply cleanly, can you please attach it?
GitHub issues do no support attachments. Anyway, it shouldn't make any difference. How did you try to apply it? Usegit apply or patch -p1.
patch -p2 < BuildBootstrap.Makefile.diff
patching file BuildBootstrap.Makefile
Hunk #1 succeeded at 19 (offset -21 lines).
Hunk #2 succeeded at 71 with fuzz 1 (offset -21 lines).
Hunk #3 succeeded at 95 (offset -21 lines).
Hunk #4 succeeded at 127 (offset -20 lines).
Hunk #5 succeeded at 149 (offset -20 lines).
Hunk #6 succeeded at 160 (offset -20 lines).
patch unexpectedly ends in middle of line
patch unexpectedly ends in middle of line
Is this patch truncated?
Ah, I guess it only applies after calling Pkg.checkout("Cxx"). If that command fails, be sure to reset the repo with git reset --hard beforehand.
I'm having a similar issue on Ubuntu Linux, did this patch end up working?
ATM I seem to be stuck:
Pkg.checkout("Cxx") ERROR: Cxx is not a git repo in checkout(::String, ::String, ::Bool, ::Bool) at ./pkg/entry.jl:225 in (::Base.Pkg.Dir.##2#3{Array{Any,1},Base.Pkg.Entry.#checkout,Tuple{String,String,Bool,Bool}})() at ./pkg/dir.jl:31 in cd(::Base.Pkg.Dir.##2#3{Array{Any,1},Base.Pkg.Entry.#checkout,Tuple{String,String,Bool,Bool}}, ::String) at ./file.jl:59 in checkout(::String) at ./pkg/pkg.jl:170 in eval_user_input(::Any, ::Base.REPL.REPLBackend) at ./REPL.jl:64 in macro expansion at ./REPL.jl:95 [inlined] in (::Base.REPL.##3#4{Base.REPL.REPLBackend})() at ./event.jl:68
On Mon, Feb 6, 2017 at 3:06 AM Walter Simson [email protected] wrote:
I'm having a similar issue on Ubuntu Linux, did this patch end up working?
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/Keno/Cxx.jl/issues/321#issuecomment-277609781, or mute the thread https://github.com/notifications/unsubscribe-auth/AAHK0AFOvdDWZOfR5FQMb0WyQqtavm8Eks5rZtSJgaJpZM4L2ZJp .
Oh, OK. I tried again:
Pkg.add("Cxx")
INFO: Cloning cache of Cxx from https://github.com/Keno/Cxx.jl.git
INFO: Installing Cxx v0.0.2
INFO: Building Cxx
writing path.jl file
Tuning for julia installation at /usr/bin with sources possibly at /usr/bin/../..
INFO: Building julia binary build
Not building release library because corresponding julia RELEASE library does not exist.
Not building debug library because corresponding julia DEBUG library does not exist.
To build, simply run the build again once the library at
To build, simply run the build again once the library at
/libjulia-debug.so
/libjulia.so
has been built.
has been built.
cc -E -P -I/usr/bin/../../src/support -I/usr/bin/../include -Isrc/clang-3.7.1/lib -Ibuild/clang-3.7.1/include -Isrc/clang-3.7.1/include -Isrc/llvm-3.7.1/include -Ibuild/llvm-3.7.1/include -DJULIA ../src/cenumvals.jl.h > build/clang_constants.jl
INFO: Package database updated
INFO: METADATA is out-of-date — you may not have the latest version of Cxx
INFO: Use `Pkg.update()` to get the latest versions of your packages
julia> Pkg.checkout("Cxx")
INFO: Checking out Cxx master...
INFO: Pulling Cxx latest master...
INFO: No packages to install, update or remove
Now at this point I applied the patch, using emacs ediff. Seems to have applied without problems.
julia> using Cxx
ERROR: LoadError: UndefVarError: TypeUtils not defined
in eval_user_input(::Any, ::Base.REPL.REPLBackend) at ./REPL.jl:64
in macro expansion at ./REPL.jl:95 [inlined]
in (::Base.REPL.##3#4{Base.REPL.REPLBackend})() at ./event.jl:68
while loading /home/nbecker/.julia/v0.5/Cxx/src/Cxx.jl, in expression starting on line 151
julia> Pkg.build("Cxx")
INFO: Building Cxx
writing path.jl file
Tuning for julia installation at /usr/bin with sources possibly at /usr/bin/../..
INFO: Building julia binary build
cd src/llvm-3.7.1 && patch -p1 < ../../llvm-patches/llvm-3.7.1.patch
can't find file to patch at input line 5
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--------------------------
|diff --git a/include/llvm/ExecutionEngine/SectionMemoryManager.h b/include/llvm/ExecutionEngine/SectionMemoryManager.h
|index 0b0dcb0..7bb96eb 100644
|--- a/include/llvm/ExecutionEngine/SectionMemoryManager.h
|+++ b/include/llvm/ExecutionEngine/SectionMemoryManager.h
--------------------------
File to patch:
Skip this patch? [y]
Skipping patch.
2 out of 2 hunks ignored
can't find file to patch at input line 46
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--------------------------
|diff --git a/include/llvm/Support/raw_ostream.h b/include/llvm/Support/raw_ostream.h
|index 28e512c..686e8c8 100644
|--- a/include/llvm/Support/raw_ostream.h
|+++ b/include/llvm/Support/raw_ostream.h
--------------------------
File to patch:
Skip this patch? [y]
Skipping patch.
3 out of 3 hunks ignored
can't find file to patch at input line 113
Perhaps...
After a failed Pkg.build/Pkg.add attempt, you need to remove ~/.julia/Cxx/deps/build/* before retrying. Probably something that could be improved too.
After cleaning things a couple of times (including .cache), then Pkg.build did download, patch, and build llvm and clang. But then restarting julia:
julia> using Cxx
ERROR: LoadError: UndefVarError: TypeUtils not defined
in eval_user_input(::Any, ::Base.REPL.REPLBackend) at ./REPL.jl:64
in macro expansion at ./REPL.jl:95 [inlined]
in (::Base.REPL.##3#4{Base.REPL.REPLBackend})() at ./event.jl:68
while loading /home/nbecker/.julia/v0.5/Cxx/src/Cxx.jl, in expression starting on line 151
Cxx master depends on the latest version of Compat.jl with was tagged three hours ago. Pkg.update should fix it.
Well it completed build, but:
using Cxx
ERROR: LoadError: LoadError: could not load library "libcxxffi"
/home/nbecker/.julia/v0.5/Cxx/src/../deps/usr/lib/libcxxffi.so: undefined symbol: _ZN5clang17ExternalASTSource19getSourceDescriptorERKNS_6ModuleE
in init_libcxxffi() at /home/nbecker/.julia/v0.5/Cxx/src/initialization.jl:21
in eval_user_input(::Any, ::Base.REPL.REPLBackend) at ./REPL.jl:64
in macro expansion at ./REPL.jl:95 [inlined]
in (::Base.REPL.##3#4{Base.REPL.REPLBackend})() at ./event.jl:68
while loading /home/nbecker/.julia/v0.5/Cxx/src/initialization.jl, in expression starting on line 23
while loading /home/nbecker/.julia/v0.5/Cxx/src/Cxx.jl, in expression starting on line 163
julia> ENV["LD_LIBRARY_PATH"]
ERROR: KeyError: key "LD_LIBRARY_PATH" not found
in eval_user_input(::Any, ::Base.REPL.REPLBackend) at ./REPL.jl:64
in macro expansion at ./REPL.jl:95 [inlined]
in (::Base.REPL.##3#4{Base.REPL.REPLBackend})() at ./event.jl:68
julia> ENV["LD_LIBRARY_PATH"]="/home/nbecker/.julia/v0.5/Cxx/deps/usr/lib"
"/home/nbecker/.julia/v0.5/Cxx/deps/usr/lib"
julia> cxx""" #include<iostream> """
ERROR: UndefVarError: @cxx_str not defined
in eval_user_input(::Any, ::Base.REPL.REPLBackend) at ./REPL.jl:64
in macro expansion at ./REPL.jl:95 [inlined]
in (::Base.REPL.##3#4{Base.REPL.REPLBackend})() at ./event.jl:68
seems there are still a couple of problems.
That's what usually happens when you get headers and binary versions crossed (e.g. because you tried to build with one julia version and load it with the other).
The build completed, but d/l and built llvm and clang in the process. Is this expected?
It do not work for me with ubuntu too. since the code base do not have clang_constants.jl anymore, at least, the line including the file should be deleted? https://github.com/Keno/Cxx.jl/blob/master/src/typetranslation.jl#L438
Julia Version 0.6.2
Commit d386e40c17 (2017-12-13 18:08 UTC)
Platform Info:
OS: Linux (x86_64-pc-linux-gnu)
CPU: Intel(R) Core(TM) i7-5820K CPU @ 3.30GHz
WORD_SIZE: 64
BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Haswell)
LAPACK: libopenblas64_
LIBM: libopenlibm
LLVM: libLLVM-3.9.1 (ORCJIT, haswell)
it was a compilation problem. after rebuilding the package by Pkg.build("Cxx"), I got some new error:
│julia> using Cxx
│INFO: Precompiling module Cxx.
│ERROR: LoadError: LoadError: could not load library "/usr/people/jingpeng/.julia/v0.6/Cxx/sr
│c/../deps/usr/lib/libcxxffi"
│/usr/people/jingpeng/.julia/v0.6/Cxx/src/../deps/usr/lib/libcxxffi.so: undefined symbol: _ZT
│VN4llvm2cl3optISsLb0ENS0_6parserISsEEEE
│Stacktrace:
│ [1] dlopen(::String, ::UInt32) at ./libdl.jl:97
│ [2] init_libcxxffi() at /usr/people/jingpeng/.julia/v0.6/Cxx/src/initialization.jl:20
│ [3] include_from_node1(::String) at ./loading.jl:576
│ [4] include(::String) at ./sysimg.jl:14
│ [5] include_from_node1(::String) at ./loading.jl:576
│ [6] include(::String) at ./sysimg.jl:14
│ [7] anonymous at ./<missing>:2
│while loading /usr/people/jingpeng/.julia/v0.6/Cxx/src/initialization.jl, in expression star
│ting on line 22
│while loading /usr/people/jingpeng/.julia/v0.6/Cxx/src/Cxx.jl, in expression starting on lin
│e 163
│ERROR: Failed to precompile Cxx to /usr/people/jingpeng/.julia/lib/v0.6/Cxx.ji.
│Stacktrace:
│ [1] compilecache(::String) at ./loading.jl:710
│ [2] _require(::Symbol) at ./loading.jl:497
│ [3] require(::Symbol) at ./loading.jl:405
Same here. Has anybody found a fix? Couldn't get it to build in 0.7-alpha, either. I could wait for that if it's easier to support and if there's an ETA.
Actually, I was able to recover: Compiling with gcc-7.3.1 instead of the system-supplied 4.8.5 seems to work.
@jstrube, how do you change which version of gcc it uses? I'm not clear on where that is defined
Either by setting the envvars before starting julia:
CC=gcc-7 CXX=g++-7 julia
or by setting them from within julia, before building Cxx
julia
ENV["CC"]="gcc-7"
ENV["CXX"]="g++-7"
Pkg.build("Cxx")
Using Cxx
In this case, my gcc executables are in the PATH, and they are called gcc-7 and g++-7`.
In addition to @nalimilan patches I also had to add the -L/usr/lib64/llvm3.9/lib/ compiler flag to make the building of libcxxffi.so and libcxxffi-debug.so succeed, adding for example
LDFLAGS += -L/usr/lib64/llvm3.9/lib
just before the respective lines in BuildBootstrap.Makefile.
