PackageCompiler.jl from scratch: running in a static environment

I'm trying to run PackageCompiler-generated binaries in a very bare environment -- a Docker image that starts with only from scratch -- and I run into a few issues before I could execute the generated app. Documenting those steps here in case we want to add a static mode (where additional libraries are bundled):

create_app("MyApp", "MyAppCompiled")

Figure out which libraries our generated binary uses, and copy those over to the lib folder:

~/PackageCompiler/examples$ ldd MyAppCompiled/bin/MyApp
        linux-vdso.so.1 (0x00007ffd86380000)
        libc.so.6 => /usr/lib/libc.so.6 (0x00007f30de3b4000)
        libdl.so.2 => /usr/lib/libdl.so.2 (0x00007f30daa10000)
        librt.so.1 => /usr/lib/librt.so.1 (0x00007f30daa05000)
        libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f30da9e3000)
        libm.so.6 => /usr/lib/libm.so.6 (0x00007f30da51f000)
        /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f30e4624000)
~/PackageCompiler/examples$ cp /lib64/ld-linux-x86-64.so.2 /usr/lib/libc.so.6 MyAppCompiled/lib
# etc

Move MyApp.so from the bin to lib folder, or Julia doesn't seem able to pick it up

Build a container:

FROM scratch
COPY MyAppCompiled /
ENV LD_LIBRARY_PATH=/lib
ENTRYPOINT ["/lib/ld-linux-x86-64.so.2"]
CMD ["/bin/MyApp"]

~/PackageCompiler/examples$ docker build . -t MyAppCompiled && docker run --rm -it MyAppCompiled
Sending build context to Docker daemon  268.1MB
Step 1/5 : FROM scratch
---> 
Step 2/5 : COPY MyAppCompiled /
---> Using cache
---> f12c51bac908
Step 3/5 : ENV LD_LIBRARY_PATH=/lib
---> Using cache
---> 44b8d2554a8f
Step 4/5 : ENTRYPOINT ["/lib/ld-linux-x86-64.so.2"]
---> Running in 1fca95357268
Removing intermediate container 1fca95357268
---> 04e51f9e29fa
Step 5/5 : CMD ["/bin/MyApp"]
---> Running in 0f57d7947861
Removing intermediate container 0f57d7947861
---> 83254aa9ba6a
Successfully built 83254aa9ba6a
Successfully tagged pkgcompiler:latest
ARGS = String[]
Base.PROGRAM_FILE = "/bin/MyApp"
DEPOT_PATH = ["//"]
LOAD_PATH = ["@"]
pwd() = "/"
Base.active_project() = nothing
Threads.nthreads() = 1
Sys.BINDIR = "/lib"
Dict{Base.PkgId,Module} with 35 entries:
CRC32c [8bf52ea8-c179-5cab-976a-9e18b702a9bc] => CRC32c
Printf [de0858da-6303-5e67-8744-51eddeeeb8d7] => Printf
LinearAlgebra [37e2e46d-f89d-539d-b4ee-838fcccc9c8e] => LinearAlgebra
UUIDs [cf7118a7-6976-5b1a-9a39-7adc72f591a4] => UUIDs
REPL [3fa0cd96-eef1-5676-8a61-b3b8758bbffb] => REPL
Libdl [8f399da3-3557-5675-b5ff-fb832c97cbdb] => Libdl
Mmap [a63ad114-7e13-5084-954f-fe012c677804] => Mmap
Example [7876af07-990d-54b4-ab0e-23690620f79a] => Example
Markdown [d6f4376e-aef5-505a-96c1-9c027394607a] => Markdown
Base64 [2a0f44e3-6c83-55bd-87e4-b1978d98bd5f] => Base64
Random [9a3f8284-a2c9-5f02-9a11-845980a1fd5c] => Random
Base [top-level] => Base
Profile [9abbd945-dff8-562f-b5e8-e1ebf5ef1b79] => Profile
HelloWorldC_jll [dca1746e-5efc-54fc-8249-22745bc95a49] => HelloWorldC_jll
Unicode [4ec0a83e-493e-50e2-b9ac-8f72acf5a8f5] => Unicode
Pkg [44cfe95a-1eb2-52ea-b672-e2afdf69b78f] => Pkg
InteractiveUtils [b77e0a4c-d291-57a0-90e8-8db25a27a240] => InteractiveUtils
SharedArrays [1a1011a3-84de-559e-8e89-a11a2f7dc383] => SharedArrays
Serialization [9e88b42a-f829-5b0c-bbe9-9e923198166b] => Serialization
MyApp [f943f3d7-887a-4ed5-b0c0-a1d6899aa8f5] => MyApp
Future [9fa8497b-333b-5362-9e8d-4d0656e87820] => Future
DelimitedFiles [8bb1440f-4735-579b-a4ab-409b98df4dab] => DelimitedFiles
SparseArrays [2f01184e-e22b-5df5-ae63-d93ebab69eaf] => SparseArrays
Dates [ade2ca70-3891-5945-98fb-dc099432e06a] => Dates
Sockets [6462fe0b-24de-5631-8697-dd941f90decc] => Sockets
Statistics [10745b16-79ce-11e8-11f9-7d13ad32a3b2] => Statistics
Core [top-level] => Core
Main [top-level] => Main
Distributed [8ba89e20-285c-5b6f-9357-94700520ee1b] => Distributed
Test [8dfed614-e22c-5e08-85e1-65c5234f0b40] => Test
Logging [56ddb016-857b-54e1-b83d-db4d58db5568] => Logging
SHA [ea8e919c-243c-51af-8825-aaa63cd721ce] => SHA
SuiteSparse [4607b0f0-06f3-5cda-b6b1-a6196a1729e9] => SuiteSparse
FileWatching [7b1f6079-737a-58dc-b8bc-7a2ca5c1b5ee] => FileWatching
LibGit2 [76f85450-5226-5b5a-8eaa-529ad045b433] => LibGit2

In this funky set-up Julia isn't able to spawn binaries (probably because it then doesn't use the dynamic linker -- unsure how that's set up by default, I haven't dug that deep):

$ docker run --rm -it MyAppCompiled
...
Running a jll package:
HelloWorld artifact at /artifacts/4da0ccfb65e328aad3d3d7f9b5c06a4564cbc53d/bin/hello_world
ERROR: IOError: could not spawn `/artifacts/4da0ccfb65e328aad3d3d7f9b5c06a4564cbc53d/bin/hello_world`: no such file or directory (ENOENT)
Stacktrace:
 [1] _spawn_primitive(::String, ::Cmd, ::Array{Any,1}) at ./process.jl:99
 [2] #585 at ./process.jl:112 [inlined]
 [3] setup_stdios(::Base.var"#585#586"{Cmd}, ::Array{Any,1}) at ./process.jl:196
 [4] _spawn at ./process.jl:111 [inlined]
 [5] run(::Cmd; wait::Bool) at ./process.jl:439
 [6] run at ./process.jl:438 [inlined]
 [7] (::MyApp.var"#1#2")(::String) at /home/tim/PackageCompiler/examples/MyApp/src/MyApp.jl:34
 [8] (::HelloWorldC_jll.var"#8#9"{MyApp.var"#1#2"})() at /home/tim/.julia/packages/HelloWorldC_jll/QWPAv/src/wrappers/x86_64-linux-gnu.jl:34
 [9] withenv(::HelloWorldC_jll.var"#8#9"{MyApp.var"#1#2"}, ::Pair{String,String}, ::Vararg{Pair{String,String},N} where N) at ./env.jl:161
 [10] hello_world(::MyApp.var"#1#2"; adjust_PATH::Bool, adjust_LIBPATH::Bool) at /home/tim/.julia/packages/HelloWorldC_jll/QWPAv/src/wrappers/x86_64-linux-gnu.jl:33
 [11] hello_world at /home/tim/.julia/packages/HelloWorldC_jll/QWPAv/src/wrappers/x86_64-linux-gnu.jl:17 [inlined]
 [12] real_main() at /home/tim/PackageCompiler/examples/MyApp/src/MyApp.jl:32
 [13] julia_main() at /home/tim/PackageCompiler/examples/MyApp/src/MyApp.jl:11
 [14] julia_main() at ./none:36

$ docker run --rm -it pkgcompiler /artifacts/4da0ccfb65e328aad3d3d7f9b5c06a4564cbc53d/bin/hello_world
Hello, World!

For debuging, you can launch docker with --env LD_DEBUG=libs since we're using the dynamic linker here.

Dec 18 '20 15:12 maleadt

Figure out which libraries our generated binary uses, and copy those over to the lib folder:

Do you think it makes sense to try to statically link those or is bundling better?

Move MyApp.so from the bin to lib folder, or Julia doesn't seem able to pick it up

That's strange. It should be set here:

https://github.com/JuliaLang/PackageCompiler.jl/blob/2d8010d8be72d65269d114b037be35081ec7f303/src/embedding_wrapper.c#L48

and just be the name of the app. From CI, it seems to find it next to the executable.

Dec 18 '20 16:12 KristofferC

Do you think it makes sense to try to statically link those or is bundling better?

Since our Yggdrasil-built artifacts are all dynamically linked, we need to bundle those libraries anyway.

Dec 18 '20 16:12 maleadt