Standard `precompile_execution_file.jl` location

[ericphanson]

It would be nice to have support for automatically picking up precompile_execution_file's. There are two sides to this as I see it.

• Package dependencies: It would be nice if there was a standard location for a precompile_execution_file such that packages could include such a file in their code and when you compile a sysimage with one of those packages PackageCompiler automatically uses that file. E.g. Plots could have such a file which just generates some plots to help alleviate TTFP, and say DifferentialEquations could have its own such file which solves some simple ODEs. So for example, PackageCompiler.create_sysimage([:Debugger, :OhMyREPL]) would use precompile_execution_files included in the code of Debugger.jl or OhMyREPL.jl if they put such a file in some standard location.
• Projects: a folder with a Manifest and Project.toml and some code which is not necessarily structured as a Julia package could also have a standard location for a precompile_execution_file. For example, a project that involves plotting differential equations could have such a file which plots a solution. That would cover code which neither Plots precompile_execution_file nor DifferentialEquations precompile_execution_file could cover (assuming they exist as in the first point), since neither depends on the other. This could also come together very smoothly for the end user if you could also do e.g. PackageCompiler.create_sysimage(sysimage_path=sysimage_path, project=env_to_precompile) which could use the Manifest and Project at env_to_precompile to get all the dependencies to include, use all their precompile_execution_files (assuming they exist), and then also use the any precompile_execution_file located at that environment.

This has come from discussion with @davidanthoff at https://github.com/julia-vscode/julia-vscode/issues/1160.

I think such a standard location could also be useful for automatically generating precompiles with SnoopCompile bot (https://github.com/timholy/SnoopCompile.jl).

[christopher-dG]

How about putting its location in Project.toml?

[ericphanson]

Would old versions of Pkg choke on parsing such Projects?

Also, I think there’s also been a lot of reluctance (reasonably, I think) on the part of the Pkg devs to put more things into the Project.toml. It seems simple enough to just choose a location and filename by convention (eg top-level, or in /test) and then we don’t need extra configuration.

Edit: I maybe was remembering some reluctance related to adding more Pkg-related config to the Project.toml as opposed to using it for non-Pkg config without needing changes in Pkg.

[christopher-dG]

On 1.0 at least, adding extra keys has no effect. I don't really understand not wanting to have anything extra in that file, because none of it has to be explicitly supported by Pkg. A feature like this would be entirely consumed by PackageCompiler. I'd much prefer to have all my project configuration centralized in the project file that in a bunch of implicit knowledge/assumptions.

[jlperla]

and then we don’t need extra configuration. Edit: I maybe was remembering some reluctance related to adding more Pkg-related config to the Project.toml as opposed to using it for non-Pkg config without needing changes in Pkg.

When I had asked people about this before @KristofferC was not explicitly opposed and package manager wouldn't trash any keys l. But this predates PackageCompilerX's coup...

In general, I think there will emerge enough variations on settings required for sysimage creation for the vscode compiler (and https://github.com/jupyter/repo2docker/issues/601 hopefully at some point!) that it would be great to have things in a reproducible location. But maybe @davidanthoff is already planning an alternative there?

The vscode package compilation tooling is amazing, so anything we can do to make package compilation environments easily reproducible (with just cloning a repo) would be very helpful for precompiling latency.

[ericphanson]

In general, I think there will emerge enough variations on settings required for sysimage creation for the vscode compiler (and jupyter/repo2docker#601 hopefully at some point!) that it would be great to have things in a reproducible location. But maybe @davidanthoff is already planning an alternative there?

Just to say, I opened this issue here because @davidanthoff thought it would be better to have the solution here instead of something VS Code specific (https://github.com/julia-vscode/julia-vscode/issues/1160#issuecomment-623104549)!

[davidanthoff]

Yeah, so my position on this is that I want to make as few calls, designs etc. that are VS Code specific. In my ideal world I just call PackageCompiler.create_sysimage(sysimage_path=sysimage_path, project=env_to_precompile) from the VS Code extension, and then it magically does everything else without the VS Code extension being in the business of locating specific files, or reading config from package files or anything like that.

[mdmaas]

Hi!

I came across this issue because I think it'd be really nice to improve the VSCode interface to PackageCompiler.

I've added a few lines of code to PackageCompiler.jl in my branch (here) so it reads this kind of config from the Project.toml. (lines 467 to 480).

Pkg has ctx.env.project.other, which seems to have room for extra config.

My Project.toml is now:

[deps]
Plots = "91a5bcdd-55d7-5caf-9e0b-520d859cae80"
[PkgCompSettings]
precompile_execution_file = "precompile_plots.jl"

and precompile_plots.jl is simply:

using Plots
p = plot(rand(5), rand(5))
display(p)

Now, when I call PackageCompiler from the REPL I can avoid passing

precompile_execution_file="precompile_plots.jl" 

and it just reads it from the Project.toml, provided that the project is the active environment.

However, when I try to build the sysimage from VSCode, it doesn't seem to be working, even if I manually activate the current directory.

@davidanthoff Is there something I'm missing about how VSCode sysimage build works? Does VSCode needs to create a "precompile_statements_file.jl" first, perhaps?

Regards,