Lazy downloading artifacts with custom sysimage

[tomyun]

Some images created by create_sysimage() tries downloading artifacts after it's loaded first time.

$ julia
julia> using PackageCompiler

julia> create_sysimage([:Gadfly], sysimage_path="gadfly.dylib")
✔ [01m:48s] PackageCompiler: compiling incremental system image

$ julia -Jgadfly.dylib
 Downloading artifact: MKL
    Downloading [================>                        ]  39.2 %

I believe this issue started happening since recent versions of PackageCompiler, presumably from 1.6 or 1.7. I'm not entirely sure if there was any relevant change in the said package, i.e. MKL.jl, triggering this issue.

I saw include_lazy_artifacts option was recently added to create_app() and create_library(), but there seems to be no equivalent option for create_sysimage().

julia> versioninfo()
Julia Version 1.6.4
Commit 35f0c911f4 (2021-11-19 03:54 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin19.5.0)
  CPU: Intel(R) Core(TM) i7-10700K CPU @ 3.80GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-11.0.1 (ORCJIT, skylake)

PackageCompiler.jl used in the example above was v2.0.2.

[KristofferC]

Hmm, but create_sysimage never bundled any artifacts?

[tomyun]

Then maybe it's happening after JuliaLinearAlgebra/MKL.jl#78 I guess. Now I have to explicitly add MKL.jl as a dependency to avoid lazy downloading and wonder if there is a better way to handle this, i.e. include_lazy_artifacts for create_sysimage().

[simonbyrne]

Ah, I just hit this: basically the problem is FFTW.jl. It has MKL_jll (for which the artifacts are lazy) in its Project.toml, but doesn't load it by default (lazy artifact downloads are triggered by __init__()). It is only loaded if you change the preferences.

However if you build FFTW.jl into your system image, it will also add MKL_jll. Since all system image __init__() statements are triggered at julia_init, starting Julia is sufficient to trigger the artifact download (even though it will never actually be used).

[simonbyrne]

The solution appears to be to use include_transitive_dependencies=false, which will avoid pulling in MKL_jll if it isn't actually loaded.

[simonbyrne]

Perhaps it is worth changing this to be the default?

[zhubonan]

I think include_transitive_dependencies=false might have some side effects. It can be useful to allow passing an array of packages that should be excluded perhaps.

[TheCedarPrince]

I had this issue as well. My only issue is I do not know if I can ignore installing the MKL dependency. Do we know how to determine if this can be ignored or not?

[zhubonan]

I had this issue as well. My only issue is I do not know if I can ignore installing the MKL dependency. Do we know how to determine if this can be ignored or not?

It depends on what you want to achieve and what the MKL_jll is used for. If you want a system image then it is harmless as wether including MKL or not in the system image should have minimum impact on the reduction of compilation delay (since MKL_jll wraps compiled MKL libraries).

If you are making an App then have the MKL_jll missing may cause problems, as a standalone App should include all dependencies in the system image. Somehow I could not get lazy aretefact download to work for App (it errors after the download, may be another issue to flag?), even it does work, it is a massive pain becuase MKL_jll aretefacts are huge.

However, if the reason of depending on MKL_jll is becuase of FFTW.jl, this is not a problem because FFTW.jl actually defaults to use FFTW_jll, and for an App the user should not switch backend anyway. In this case, my solution was to make a fork of FFTW.jl without MKL_jll as dependency, and include this forked version in the App instead.