Compiled libraries depend on being inside the "bin"-folder

[ThummeTo]

Hi,

I just found out, that the compiled DLLs rely on beeing inside a folder called "bin" (as they are after the compilation process). Renaming the "bin"-folder after compilation (like to "binary" for example) results in the following error:

ERROR: Unable to load dependent library ...\binary\../bin/libgcc_s_seh-1.dll

There should be no path construct like that in the final DLL (go one path element down an re-enter the "bin"-path), a DLL should be independent of its path (if possible). The "bin"-folder can be moved to any location and the DLLs still work, as long the "bin"-folder is not renamed.

BTW, the EXE fails with code 0xc0000142.

Thanks and regards!

Edit: Testet on Windows 10 (64-bit) with the shipped library example.

[ThummeTo]

Steps to reproduce:

• Compile the library example.
• Try to load the DLL (success).
• Rename folder "bin" to whatever you want.
• Try to load the DLL (failure).
• Rename the folder to "bin" again.
• Try to load the DLL (success).

Best regards!

[KristofferC]

@staticfloat I remember you saying that this is set during build time and there isn't an easy way to change it. Perhaps you could elaborate on it a bit here?

[ThummeTo]

As an alternative/quick fix: Would it be possible to specify a custom "bin"-directory name before compilation start? Maybe a keyword argument for create_library?

[staticfloat]

This is because the julia executable needs to know the relative path to some of these binaries at buildtime. We currently 'hardcode" these here: https://github.com/JuliaLang/julia/blob/fd8b2abb0bea04799204bfa5c95539544dc35cf3/Make.inc#L1504-L1516

It can be altered after the fact; you can use the stringreplace binary that Julia builds to patch out the string (which looks something like "../bin/libfoo.dll:../bin/libbar.dll:...") and replace it with something else, but you need to be careful that your new name doesn't increase the string length beyond the maximum allowed, currently 1024 bytes.

[ThummeTo]

Thanks for the reply.

Can you please give me a hint on when (in the packagecompiler-process) and how to use this stringreplace binary to patch out the strings?

[KristofferC]

This is because the julia executable needs to know the relative path to some of these binaries at buildtime.

But in Windows the julia executable and the libraries are in the same folder (bin) so is this really needed there?

[staticfloat]

But in Windows the julia executable and the libraries are in the same folder (bin) so is this really needed there?

This was added as a stepping stone toward having the DLLs exist within an stdlib artifacts directory. So the raw capability is needed, despite it not being used to its full potential, yet.

Can you please give me a hint on when (in the packagecompiler-process) and how to use this stringreplace binary to patch out the strings?

Once you've changed the bin folder's name, you will need to patch libjulia.dll. I actually have some Julia code that does the same thing as what stringreplace did:

#!/usr/bin/env julia

if length(ARGS) != 1
    error("Usage: $(@__FILE__) <path_to_libjulia.dll>")
end

libjulia_path = ARGS[1]

if !isfile(libjulia_path)
    error("Unable to open libjulia.dll at $(libjulia_path)")
end

open(libjulia_path, read=true, write=true) do io
    # Search for `../bin/` string:
    needle = "../bin/"
    readuntil(io, needle)
    skip(io, -length(needle))
    libpath_offset = position(io)

    libpath = split(String(readuntil(io, UInt8(0))), ":")
    @info("Found embedded libpath", libpath, libpath_offset)

    # Get rid of `../bin/` prefix:
    libpath = map(libpath) do l
        if startswith(l, "../bin/")
            return l[8:end]
        end
    end

    @info("Filtered libpath", libpath)

    # Write filtered libpath out to the file, terminate with NULL.
    seek(io, libpath_offset)
    write(io, join(libpath, ":"))
    write(io, UInt8(0))
end

That should allow you to rewrite the libpaths that look like "../bin/libfoo.dll:../bin/libbar.dll" to instead just read "libfoo.dll:libbar.dll", etc...

[ThummeTo]

Thank you very much, I will give it a try and report!

EDIT: This works excellent!

[reyntjesr]

I also ran into this problem and posted a stackoverflow question before I found this issue. I compiled the simple integer increment demonstration of the help pages and have written a small console program in the c# language that makes use of the compiled library. I changed the build.jl file of the example library to make things work. Compared with the code above I added an extra return l. I also changed the write() function calls to Base.write() because there's already a write boolean value in the function.

using PackageCompiler

target_dir = get(ENV, "OUTDIR", "$(@__DIR__)/../../MyLibCompiled")
target_dir = replace(target_dir, "\\" => "/")       # Change Windows paths to use "/"

println("Creating library in $target_dir")
PackageCompiler.create_library(".", target_dir;
    lib_name="libinc",
    precompile_execution_file=["$(@__DIR__)/generate_precompile.jl"],
    precompile_statements_file=["$(@__DIR__)/additional_precompile.jl"],
    incremental=false,
    filter_stdlibs=true,
    header_files=["$(@__DIR__)/mylib.h"]
)

libjulia_path = joinpath(target_dir,"bin/libjulia.dll")
libjulia_path = replace(libjulia_path, "\\" => "/")       # Change Windows paths to use "/"

if !isfile(libjulia_path)
    error("Unable to open libjulia.dll at $(libjulia_path)")
end

open(libjulia_path, read=true, write=true) do io
    # Search for ../bin/ string:
    needle = "../bin/"
    readuntil(io, needle)
    skip(io, -length(needle))
    libpath_offset = position(io)

    libpath = split(String(readuntil(io, UInt8(0))), ":")
    @info("Found embedded libpath", libpath, libpath_offset)

    # Get rid of `../bin/` prefix:
    libpath = map(libpath) do l
        if startswith(l, "../bin/")
            return l[8:end]
        elseif startswith(l, "@../bin/")
            return "@" * l[9:end]
        else
            return l
       end
    end

    @info("Filtered libpath", libpath)

    # Write filtered libpath out to the file, terminate with NULL.
    seek(io, libpath_offset)
    libspath = join(libpath, ":")
    Base.write(io, libspath)
    Base.write(io, UInt8(0))
end

It would be nice if you can remove these ../bin references with an extra create_library parameter instead of changing the existing libjulia.dll file.