MLJModels.jl icon indicating copy to clipboard operation
MLJModels.jl copied to clipboard

Error needing catching in `@iload` ?

Open ablaom opened this issue 4 years ago • 9 comments

It seems there is an error that we are not catching (and ignoring) here, as the interactive part following the message works fine:

using MLJ # in a new environment with only MLJ
julia> Clf = @iload EvoTreeClassifier
[ Info: For silent loading, specify `verbosity=0`. 
[ Info: Precompiling EvoTrees [f6006082-12f8-11e9-0c9c-0d5d367ab1e5]
ERROR: LoadError: ArgumentError: Package CUDA [052768ef-5323-5732-b1bb-66c8b64840ba] is required but does not seem to be installed:
 - Run `Pkg.instantiate()` to install all recorded dependencies.

Stacktrace:
 [1] _require(pkg::Base.PkgId)
   @ Base ./loading.jl:990
 [2] require(uuidkey::Base.PkgId)
   @ Base ./loading.jl:914
 [3] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:901
 [4] include
   @ ./Base.jl:386 [inlined]
 [5] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, 
dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::Nothing)                                                
   @ Base ./loading.jl:1213
 [6] top-level scope
   @ none:1
 [7] eval
   @ ./boot.jl:360 [inlined]
 [8] eval(x::Expr)
   @ Base.MainInclude ./client.jl:446
 [9] top-level scope
   @ none:1
in expression starting at /Users/anthony/Dropbox/Julia7/EvoTrees/src/EvoTrees.jl:1

The package providing an interface to `EvoTreeClassifier` is not in your current environment.
What do you want to do? 

 > Install EvoTrees in current environment.
   Abort.

ablaom avatar May 24 '21 22:05 ablaom

Maybe related to https://github.com/alan-turing-institute/MLJ.jl/issues/757

ablaom avatar May 24 '21 22:05 ablaom

cc @giordano

ablaom avatar May 24 '21 22:05 ablaom

Given the issue you're experiencing it's likely there is an exception that isn't being caught, but I can't reproduce your issue in a clean depot:

% JULIA_PKG_SERVER="pkg.julialang.org" JULIA_DEPOT_PATH=$(mktemp -d) julia -q
(@v1.6) pkg> add MLJ#dev
  Installing known registries into `/tmp/tmp.KauxRMUxvt`

       Added registry `General` to `/tmp/tmp.KauxRMUxvt/registries/General`
     Cloning git-repo `https://github.com/alan-turing-institute/MLJ.jl.git`
    Updating git-repo `https://github.com/alan-turing-institute/MLJ.jl.git`
    Updating registry at `/tmp/tmp.KauxRMUxvt/registries/General`
   Resolving package versions...
   Installed ProgressMeter ─────────────── v1.6.2
   Installed StatisticalTraits ─────────── v1.1.0
   Installed IniFile ───────────────────── v0.5.0
   Installed RecipesBase ───────────────── v1.1.1
   Installed Formatting ────────────────── v0.4.2
   Installed FixedPointNumbers ─────────── v0.8.4
   Installed Libiconv_jll ──────────────── v1.16.1+0
   Installed Distances ─────────────────── v0.10.3
   Installed HTTP ──────────────────────── v0.9.9
   Installed MLJBase ───────────────────── v0.18.7
   Installed MLJTuning ─────────────────── v0.6.5
   Installed Compat ────────────────────── v3.30.0
   Installed Parsers ───────────────────── v1.1.0
   Installed MLJOpenML ─────────────────── v1.0.0
   Installed Parameters ────────────────── v0.12.2
   Installed XML2_jll ──────────────────── v2.9.12+0
   Installed Preferences ───────────────── v1.2.2
   Installed Reexport ──────────────────── v1.0.0
   Installed IteratorInterfaceExtensions ─ v1.0.0
   Installed EzXML ─────────────────────── v1.1.0
   Installed MLJSerialization ──────────── v1.1.2
   Installed Requires ──────────────────── v1.1.3
   Installed Memento ───────────────────── v1.1.2
   Installed DataAPI ───────────────────── v1.6.0
   Installed ChainRulesCore ────────────── v0.9.44
   Installed MLJModels ─────────────────── v0.14.6
   Installed Distributions ─────────────── v0.24.18
   Installed JSON ──────────────────────── v0.21.1
   Installed Rmath_jll ─────────────────── v0.3.0+0
   Installed FilePathsBase ─────────────── v0.9.10
   Installed Rmath ─────────────────────── v0.7.0
   Installed LearnBase ─────────────────── v0.4.1
   Installed BSON ──────────────────────── v0.3.3
   Installed DocStringExtensions ───────── v0.8.4
   Installed MLJIteration ──────────────── v0.3.0
   Installed Crayons ───────────────────── v4.0.4
   Installed DataValueInterfaces ───────── v1.0.0
   Installed ColorTypes ────────────────── v0.11.0
   Installed JLSO ──────────────────────── v2.5.0
   Installed QuadGK ────────────────────── v2.4.1
   Installed MLJModelInterface ─────────── v1.1.0
   Installed MLJScientificTypes ────────── v0.4.7
   Installed SortingAlgorithms ─────────── v1.0.0
   Installed URIs ──────────────────────── v1.3.0
   Installed TableTraits ───────────────── v1.0.1
   Installed StatsAPI ──────────────────── v1.0.0
   Installed StatsBase ─────────────────── v0.33.8
   Installed IterationControl ──────────── v0.4.0
   Installed LogExpFunctions ───────────── v0.2.4
   Installed ComputationalResources ────── v0.3.2
   Installed Missings ──────────────────── v1.0.0
   Installed StableRNGs ────────────────── v1.0.0
   Installed UnPack ────────────────────── v1.0.2
   Installed DataStructures ────────────── v0.18.9
   Installed Tables ────────────────────── v1.4.2
   Installed CodecZlib ─────────────────── v0.7.0
   Installed ScientificTypes ───────────── v1.1.2
   Installed SpecialFunctions ──────────── v1.4.1
   Installed EarlyStopping ─────────────── v0.1.8
   Installed Syslogs ───────────────────── v0.3.0
   Installed JLLWrappers ───────────────── v1.3.0
   Installed ExprTools ─────────────────── v0.1.3
   Installed StructTypes ───────────────── v1.7.2
   Installed OpenSpecFun_jll ───────────── v0.5.4+0
   Installed InvertedIndices ───────────── v1.0.0
   Installed Mocking ───────────────────── v0.7.1
   Installed CategoricalArrays ─────────── v0.10.0
   Installed PDMats ────────────────────── v0.11.0
   Installed TranscodingStreams ────────── v0.9.5
   Installed StatsFuns ─────────────────── v0.9.8
   Installed PersistenceDiagramsBase ───── v0.1.1
   Installed TimeZones ─────────────────── v1.5.5
   Installed FillArrays ────────────────── v0.11.7
   Installed OrderedCollections ────────── v1.4.1
   Installed MbedTLS ───────────────────── v1.0.3
   Installed LossFunctions ─────────────── v0.7.0
   Installed PrettyTables ──────────────── v1.0.1
   Installed LatinHypercubeSampling ────── v1.8.0
  Downloaded artifact: Libiconv
  Downloaded artifact: Libiconv
  Downloaded artifact: XML2
  Downloaded artifact: XML2
  Downloaded artifact: Rmath
  Downloaded artifact: OpenSpecFun
    Updating `/tmp/tmp.KauxRMUxvt/environments/v1.6/Project.toml`
  [add582a8] + MLJ v0.16.4 `https://github.com/alan-turing-institute/MLJ.jl.git#dev`
    Updating `/tmp/tmp.KauxRMUxvt/environments/v1.6/Manifest.toml`
  [fbb218c0] + BSON v0.3.3
  [324d7699] + CategoricalArrays v0.10.0
  [d360d2e6] + ChainRulesCore v0.9.44
  [944b1d66] + CodecZlib v0.7.0
  [3da002f7] + ColorTypes v0.11.0
  [34da2185] + Compat v3.30.0
  [ed09eef8] + ComputationalResources v0.3.2
  [a8cc5b0e] + Crayons v4.0.4
  [9a962f9c] + DataAPI v1.6.0
  [864edb3b] + DataStructures v0.18.9
  [e2d170a0] + DataValueInterfaces v1.0.0
  [b4f34e82] + Distances v0.10.3
  [31c24e10] + Distributions v0.24.18
  [ffbed154] + DocStringExtensions v0.8.4
  [792122b4] + EarlyStopping v0.1.8
  [e2ba6199] + ExprTools v0.1.3
  [8f5d6c58] + EzXML v1.1.0
  [48062228] + FilePathsBase v0.9.10
  [1a297f60] + FillArrays v0.11.7
  [53c48c17] + FixedPointNumbers v0.8.4
  [59287772] + Formatting v0.4.2
  [cd3eb016] + HTTP v0.9.9
  [83e8ac13] + IniFile v0.5.0
  [41ab1584] + InvertedIndices v1.0.0
  [b3c1a2ee] + IterationControl v0.4.0
  [82899510] + IteratorInterfaceExtensions v1.0.0
  [692b3bcd] + JLLWrappers v1.3.0
  [9da8a3cd] + JLSO v2.5.0
  [682c06a0] + JSON v0.21.1
  [a5e1c1ea] + LatinHypercubeSampling v1.8.0
  [7f8f8fb0] + LearnBase v0.4.1
  [2ab3a3ac] + LogExpFunctions v0.2.4
  [30fc2ffe] + LossFunctions v0.7.0
  [add582a8] + MLJ v0.16.4 `https://github.com/alan-turing-institute/MLJ.jl.git#dev`
  [a7f614a8] + MLJBase v0.18.7
  [614be32b] + MLJIteration v0.3.0
  [e80e1ace] + MLJModelInterface v1.1.0
  [d491faf4] + MLJModels v0.14.6
  [cbea4545] + MLJOpenML v1.0.0
  [2e2323e0] + MLJScientificTypes v0.4.7
  [17bed46d] + MLJSerialization v1.1.2
  [03970b2e] + MLJTuning v0.6.5
  [739be429] + MbedTLS v1.0.3
  [f28f55f0] + Memento v1.1.2
  [e1d29d7a] + Missings v1.0.0
  [78c3b35d] + Mocking v0.7.1
  [bac558e1] + OrderedCollections v1.4.1
  [90014a1f] + PDMats v0.11.0
  [d96e819e] + Parameters v0.12.2
  [69de0a69] + Parsers v1.1.0
  [b1ad91c1] + PersistenceDiagramsBase v0.1.1
  [21216c6a] + Preferences v1.2.2
  [08abe8d2] + PrettyTables v1.0.1
  [92933f4c] + ProgressMeter v1.6.2
  [1fd47b50] + QuadGK v2.4.1
  [3cdcf5f2] + RecipesBase v1.1.1
  [189a3867] + Reexport v1.0.0
  [ae029012] + Requires v1.1.3
  [79098fc4] + Rmath v0.7.0
  [321657f4] + ScientificTypes v1.1.2
  [a2af1166] + SortingAlgorithms v1.0.0
  [276daf66] + SpecialFunctions v1.4.1
  [860ef19b] + StableRNGs v1.0.0
  [64bff920] + StatisticalTraits v1.1.0
  [82ae8749] + StatsAPI v1.0.0
  [2913bbd2] + StatsBase v0.33.8
  [4c63d2b9] + StatsFuns v0.9.8
  [856f2bd8] + StructTypes v1.7.2
  [cea106d9] + Syslogs v0.3.0
  [3783bdb8] + TableTraits v1.0.1
  [bd369af6] + Tables v1.4.2
  [f269a46b] + TimeZones v1.5.5
  [3bb67fe8] + TranscodingStreams v0.9.5
  [5c2747f8] + URIs v1.3.0
  [3a884ed6] + UnPack v1.0.2
  [94ce4f54] + Libiconv_jll v1.16.1+0
  [efe28fd5] + OpenSpecFun_jll v0.5.4+0
  [f50d1b31] + Rmath_jll v0.3.0+0
  [02c8fc9c] + XML2_jll v2.9.12+0
  [0dad84c5] + ArgTools
  [56f22d72] + Artifacts
  [2a0f44e3] + Base64
  [ade2ca70] + Dates
  [8bb1440f] + DelimitedFiles
  [8ba89e20] + Distributed
  [f43a241f] + Downloads
  [9fa8497b] + Future
  [b77e0a4c] + InteractiveUtils
  [4af54fe1] + LazyArtifacts
  [b27032c2] + LibCURL
  [76f85450] + LibGit2
  [8f399da3] + Libdl
  [37e2e46d] + LinearAlgebra
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [a63ad114] + Mmap
  [ca575930] + NetworkOptions
  [44cfe95a] + Pkg
  [de0858da] + Printf
  [3fa0cd96] + REPL
  [9a3f8284] + Random
  [ea8e919c] + SHA
  [9e88b42a] + Serialization
  [1a1011a3] + SharedArrays
  [6462fe0b] + Sockets
  [2f01184e] + SparseArrays
  [10745b16] + Statistics
  [4607b0f0] + SuiteSparse
  [fa267f1f] + TOML
  [a4e569a6] + Tar
  [8dfed614] + Test
  [cf7118a7] + UUIDs
  [4ec0a83e] + Unicode
  [e66e0078] + CompilerSupportLibraries_jll
  [deac9b47] + LibCURL_jll
  [29816b5a] + LibSSH2_jll
  [c8ffd9c3] + MbedTLS_jll
  [14a3606d] + MozillaCACerts_jll
  [83775a58] + Zlib_jll
  [8e850ede] + nghttp2_jll
  [3f19e933] + p7zip_jll
    Building TimeZones → `/tmp/tmp.KauxRMUxvt/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/960099aed321e05ac649c90d583d59c9309faee1/build.log`
Precompiling project...
  80 dependencies successfully precompiled in 39 seconds (3 already precompiled)

(@v1.6) pkg> 

julia> using MLJ

julia> Clf = @iload EvoTreeClassifier
[ Info: For silent loading, specify `verbosity=0`. 

The package providing an interface to `EvoTreeClassifier` is not in your current environment.
What do you want to do? 

 > Install EvoTrees in current environment.
   Abort.

   Resolving package versions...
   Installed CEnum ────────── v0.4.1
   Installed BFloat16s ────── v0.1.0
   Installed EarCut_jll ───── v2.1.5+1
   Installed NetworkLayout ── v0.3.0
   Installed MacroTools ───── v0.5.6
   Installed Scratch ──────── v1.0.3
   Installed RandomNumbers ── v1.4.0
   Installed IterTools ────── v1.3.0
   Installed AbstractFFTs ─── v1.0.1
   Installed GeometryBasics ─ v0.3.12
   Installed StructArrays ─── v0.5.1
   Installed Memoize ──────── v0.4.4
   Installed GPUCompiler ──── v0.11.5
   Installed Adapt ────────── v3.3.0
   Installed Random123 ────── v1.3.1
   Installed LLVM ─────────── v3.7.1
   Installed StaticArrays ─── v1.2.0
   Installed TimerOutputs ─── v0.5.9
   Installed GPUArrays ────── v6.4.1
   Installed EvoTrees ─────── v0.8.0
   Installed CUDA ─────────── v3.2.1
  Downloaded artifact: EarCut
    Updating `/tmp/tmp.KauxRMUxvt/environments/v1.6/Project.toml`
  [f6006082] + EvoTrees v0.8.0
    Updating `/tmp/tmp.KauxRMUxvt/environments/v1.6/Manifest.toml`
  [621f4979] + AbstractFFTs v1.0.1
  [79e6a3ab] + Adapt v3.3.0
  [ab4f0b2a] + BFloat16s v0.1.0
  [fa961155] + CEnum v0.4.1
  [052768ef] + CUDA v3.2.1
  [f6006082] + EvoTrees v0.8.0
  [0c68f7d7] + GPUArrays v6.4.1
  [61eb1bfa] + GPUCompiler v0.11.5
  [5c1252a2] + GeometryBasics v0.3.12
  [c8e1da08] + IterTools v1.3.0
  [929cbde3] + LLVM v3.7.1
  [1914dd2f] + MacroTools v0.5.6
  [c03570c3] + Memoize v0.4.4
  [46757867] + NetworkLayout v0.3.0
  [74087812] + Random123 v1.3.1
  [e6cf234a] + RandomNumbers v1.4.0
  [6c6a2e73] + Scratch v1.0.3
  [90137ffa] + StaticArrays v1.2.0
  [09ab397b] + StructArrays v0.5.1
  [a759f4b9] + TimerOutputs v0.5.9
  [5ae413db] + EarCut_jll v2.1.5+1
    Building Random123 → `/tmp/tmp.KauxRMUxvt/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/7c6710c8198fd4444b5eb6a3840b7d47bd3593c5/build.log`
Precompiling project...
  21 dependencies successfully precompiled in 45 seconds (83 already precompiled)
import EvoTrees ✔
EvoTrees.EvoTreeClassifier

julia>

giordano avatar May 25 '21 14:05 giordano

Reading the code, I'm having a hard time figuring out where the exception is being thrown. Can you please show the output of @macroexpand @iload EvoTreeClassifier for you? I get

quote
    #= /tmp/tmp.O88ehAk8Zj/packages/MLJModels/9mQfs/src/loading.jl:129 =#
    begin
        #= /tmp/tmp.O88ehAk8Zj/packages/MLJModels/9mQfs/src/loading.jl:168 =#
        true && begin
            #= logging.jl:350 =#
            let
                #= logging.jl:351 =#
                var"#47#level" = Base.CoreLogging.Info
                #= logging.jl:352 =#
                var"#48#std_level" = Base.CoreLogging.convert(Base.CoreLogging.LogLevel, var"#47#level")
                #= logging.jl:353 =#
                if var"#48#std_level" >= Base.CoreLogging._min_enabled_level[]
                    #= logging.jl:354 =#
                    var"#49#group" = :loading
                    #= logging.jl:355 =#
                    var"#50#_module" = Main
                    #= logging.jl:356 =#
                    var"#51#logger" = Base.CoreLogging.current_logger_for_env(var"#48#std_level", var"#49#group", var"#50#_module")
                    #= logging.jl:357 =#
                    if !(var"#51#logger" === Base.CoreLogging.nothing)
                        #= logging.jl:358 =#
                        var"#52#id" = :Main_75496672
                        #= logging.jl:361 =#
                        if Base.CoreLogging._invoked_shouldlog(var"#51#logger", var"#47#level", var"#50#_module", var"#49#group", var"#52#id")
                            #= logging.jl:362 =#
                            var"#53#file" = "/tmp/tmp.O88ehAk8Zj/packages/MLJModels/9mQfs/src/loading.jl"
                            #= logging.jl:363 =#
                            var"#54#line" = 168
                            #= logging.jl:364 =#
                            local var"#55#msg", var"#56#kwargs"
                            #= logging.jl:365 =#
                            begin
                                #= logging.jl:308 =#
                                var"#55#msg" = "For silent loading, specify `verbosity=0`. "
                                #= logging.jl:309 =#
                                var"#56#kwargs" = (;)
                                #= logging.jl:310 =#
                                true
                            end && Base.CoreLogging.handle_message(var"#51#logger", var"#47#level", var"#55#msg", var"#50#_module", var"#49#group", var"#52#id", var"#53#file", var"#54#line"; var"#56#kwargs"...)
                        end
                    end
                end
                #= logging.jl:371 =#
                Base.CoreLogging.nothing
            end
        end
        #= /tmp/tmp.O88ehAk8Zj/packages/MLJModels/9mQfs/src/loading.jl:169 =#
        var"#37#proxy" = (MLJModels.MLJModels).info("EvoTreeClassifier"; pkg = nothing, interactive = true)
        #= /tmp/tmp.O88ehAk8Zj/packages/MLJModels/9mQfs/src/loading.jl:172 =#
        var"#38#handle" = (MLJModels.MLJModels).Handle((var"#37#proxy").name, (var"#37#proxy").package_name)
        #= /tmp/tmp.O88ehAk8Zj/packages/MLJModels/9mQfs/src/loading.jl:173 =#
        var"#39#dic" = (MLJModels.MLJModels).INFO_GIVEN_HANDLE[var"#38#handle"]
        #= /tmp/tmp.O88ehAk8Zj/packages/MLJModels/9mQfs/src/loading.jl:174 =#
        (var"#40#path_str", var"#41#pkg_str") = (var"#39#dic"[:load_path], var"#39#dic"[:package_name])
        #= /tmp/tmp.O88ehAk8Zj/packages/MLJModels/9mQfs/src/loading.jl:175 =#
        var"#42#path" = var"#40#path_str" |> (MLJModels.Meta).parse
        #= /tmp/tmp.O88ehAk8Zj/packages/MLJModels/9mQfs/src/loading.jl:176 =#
        var"#43#pkg" = MLJModels.Symbol(var"#41#pkg_str")
    end
    begin
        #= /tmp/tmp.O88ehAk8Zj/packages/MLJModels/9mQfs/src/loading.jl:181 =#
        var"#44#path_components" = MLJModels.Symbol.(MLJModels.split(var"#40#path_str", '.'))
        #= /tmp/tmp.O88ehAk8Zj/packages/MLJModels/9mQfs/src/loading.jl:184 =#
        var"#45#api_pkg" = var"#44#path_components"[1]
        #= /tmp/tmp.O88ehAk8Zj/packages/MLJModels/9mQfs/src/loading.jl:185 =#
        var"#46#api_pkg_str" = MLJModels.string(var"#45#api_pkg")
        #= /tmp/tmp.O88ehAk8Zj/packages/MLJModels/9mQfs/src/loading.jl:187 =#
        if false || true
            #= /tmp/tmp.O88ehAk8Zj/packages/MLJModels/9mQfs/src/loading.jl:188 =#
            try
                #= /tmp/tmp.O88ehAk8Zj/packages/MLJModels/9mQfs/src/loading.jl:189 =#
                (MLJModels.MLJModels)._import(Main, var"#45#api_pkg", var"#43#pkg", false)
            catch
                #= /tmp/tmp.O88ehAk8Zj/packages/MLJModels/9mQfs/src/loading.jl:191 =#
                if true
                    #= /tmp/tmp.O88ehAk8Zj/packages/MLJModels/9mQfs/src/loading.jl:192 =#
                    (MLJModels.MLJModels).request("The package providing an interface " * "to `EvoTreeClassifier` " * "is not in your " * "current environment.\n" * "What do you want to do? ", "Install $(var"#46#api_pkg_str") in current environment.", "Abort.") == 1 || MLJModels.throw(MLJModels.InterruptException)
                end
                #= /tmp/tmp.O88ehAk8Zj/packages/MLJModels/9mQfs/src/loading.jl:202 =#
                (MLJModels.MLJModels)._import(Main, :Pkg, :Pkg, false)
                #= /tmp/tmp.O88ehAk8Zj/packages/MLJModels/9mQfs/src/loading.jl:203 =#
                (MLJModels.Pkg).add(var"#46#api_pkg_str")
            end
        end
        #= /tmp/tmp.O88ehAk8Zj/packages/MLJModels/9mQfs/src/loading.jl:206 =#
        (MLJModels.MLJModels)._import(Main, var"#45#api_pkg", var"#43#pkg", true)
        #= /tmp/tmp.O88ehAk8Zj/packages/MLJModels/9mQfs/src/loading.jl:207 =#
        (MLJModels.MLJModels)._eval(Main, var"#42#path")
    end
end

Trying to load the package before the interactive Pkg prompt happens only at line L189 https://github.com/alan-turing-institute/MLJModels.jl/blob/6722abd5546d1cd58ff9547e98c7bdcf60ffcfb5/src/loading.jl#L188-L190 which is protected in a try/catch block and isn't rethrown nor shown

giordano avatar May 25 '21 16:05 giordano

@giordano Thanks for investigating. Naturally this is difficult to debug without being able to reproduce the error. Let me make sure I can also reproduce and get back to you. I actually don't think I was using a fresh environment after all. My apologies for not being more thorough.

ablaom avatar May 25 '21 21:05 ablaom

Okay, I do get (a different) error in a bare environment.

The error

julia> using MLJModels
julia> @iload EvoTreeRegressor
[ Info: For silent loading, specify `verbosity=0`. 
[ Info: Precompiling EvoTrees [f6006082-12f8-11e9-0c9c-0d5d367ab1e5]
ERROR: LoadError: ArgumentError: Package StaticArrays [90137ffa-7385-5640-81b9-e52037218182] is required but does not seem to be installed:
 - Run `Pkg.instantiate()` to install all recorded dependencies.

Stacktrace:
 [1] _require(pkg::Base.PkgId)
   @ Base ./loading.jl:990
 [2] require(uuidkey::Base.PkgId)
   @ Base ./loading.jl:914
 [3] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:901
 [4] include
   @ ./Base.jl:386 [inlined]
 [5] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, 
dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::Nothing)                                                
   @ Base ./loading.jl:1213
 [6] top-level scope
   @ none:1
 [7] eval
   @ ./boot.jl:360 [inlined]
 [8] eval(x::Expr)
   @ Base.MainInclude ./client.jl:446
 [9] top-level scope
   @ none:1
in expression starting at /Users/anthony/Dropbox/Julia7/EvoTrees/src/EvoTrees.jl:1

The package providing an interface to `EvoTreeRegressor` is not in your current environment.
What do you want to do? 

   Install EvoTrees in current environment.
 > Abort.

My environment

(@testload) pkg> st
      Status `~/.julia/environments/testload/Project.toml`
  [d491faf4] MLJModels v0.14.6
(@testload) pkg> st -m
      Status `~/.julia/environments/testload/Manifest.toml`
  [324d7699] CategoricalArrays v0.10.0
  [d360d2e6] ChainRulesCore v0.9.44
  [3da002f7] ColorTypes v0.11.0
  [34da2185] Compat v3.30.0
  [ed09eef8] ComputationalResources v0.3.2
  [a8cc5b0e] Crayons v4.0.4
  [9a962f9c] DataAPI v1.6.0
  [864edb3b] DataStructures v0.18.9
  [e2d170a0] DataValueInterfaces v1.0.0
  [b4f34e82] Distances v0.10.3
  [31c24e10] Distributions v0.25.1
  [ffbed154] DocStringExtensions v0.8.4
  [1a297f60] FillArrays v0.11.7
  [53c48c17] FixedPointNumbers v0.8.4
  [59287772] Formatting v0.4.2
  [41ab1584] InvertedIndices v1.0.0
  [82899510] IteratorInterfaceExtensions v1.0.0
  [692b3bcd] JLLWrappers v1.3.0
  [682c06a0] JSON v0.21.1
  [7f8f8fb0] LearnBase v0.4.1
  [2ab3a3ac] LogExpFunctions v0.2.4
  [30fc2ffe] LossFunctions v0.7.0
  [a7f614a8] MLJBase v0.18.7
  [e80e1ace] MLJModelInterface v1.1.0
  [d491faf4] MLJModels v0.14.6
  [2e2323e0] MLJScientificTypes v0.4.7
  [e1d29d7a] Missings v1.0.0
  [bac558e1] OrderedCollections v1.4.1
  [90014a1f] PDMats v0.11.0
  [d96e819e] Parameters v0.12.2
  [69de0a69] Parsers v1.1.0
  [b1ad91c1] PersistenceDiagramsBase v0.1.1
  [21216c6a] Preferences v1.2.2
  [08abe8d2] PrettyTables v1.0.1
  [92933f4c] ProgressMeter v1.6.2
  [1fd47b50] QuadGK v2.4.1
  [3cdcf5f2] RecipesBase v1.1.1
  [189a3867] Reexport v1.0.0
  [ae029012] Requires v1.1.3
  [79098fc4] Rmath v0.7.0
  [321657f4] ScientificTypes v1.1.2
  [a2af1166] SortingAlgorithms v1.0.0
  [276daf66] SpecialFunctions v1.4.1
  [64bff920] StatisticalTraits v1.1.0
  [82ae8749] StatsAPI v1.0.0
  [2913bbd2] StatsBase v0.33.8
  [4c63d2b9] StatsFuns v0.9.8
  [856f2bd8] StructTypes v1.7.2
  [3783bdb8] TableTraits v1.0.1
  [bd369af6] Tables v1.4.2
  [3a884ed6] UnPack v1.0.2
  [efe28fd5] OpenSpecFun_jll v0.5.4+0
  [f50d1b31] Rmath_jll v0.3.0+0
  [0dad84c5] ArgTools
  [56f22d72] Artifacts
  [2a0f44e3] Base64
  [ade2ca70] Dates
  [8bb1440f] DelimitedFiles
  [8ba89e20] Distributed
  [f43a241f] Downloads
  [9fa8497b] Future
  [b77e0a4c] InteractiveUtils
  [b27032c2] LibCURL
  [76f85450] LibGit2
  [8f399da3] Libdl
  [37e2e46d] LinearAlgebra
  [56ddb016] Logging
  [d6f4376e] Markdown
  [a63ad114] Mmap
  [ca575930] NetworkOptions
  [44cfe95a] Pkg
  [de0858da] Printf
  [3fa0cd96] REPL
  [9a3f8284] Random
  [ea8e919c] SHA
  [9e88b42a] Serialization
  [1a1011a3] SharedArrays
  [6462fe0b] Sockets
  [2f01184e] SparseArrays
  [10745b16] Statistics
  [4607b0f0] SuiteSparse
  [fa267f1f] TOML
  [a4e569a6] Tar
  [8dfed614] Test
  [cf7118a7] UUIDs
  [4ec0a83e] Unicode
  [e66e0078] CompilerSupportLibraries_jll
  [deac9b47] LibCURL_jll
  [29816b5a] LibSSH2_jll
  [c8ffd9c3] MbedTLS_jll
  [14a3606d] MozillaCACerts_jll
  [83775a58] Zlib_jll
  [8e850ede] nghttp2_jll
  [3f19e933] p7zip_jll

The macro expansion


julia> @macroexpand @iload EvoTreeRegressor
quote
    #= /Users/anthony/.julia/packages/MLJModels/9mQfs/src/loading.jl:129 =#
    begin
        #= /Users/anthony/.julia/packages/MLJModels/9mQfs/src/loading.jl:168 =#
        true && begin
                #= logging.jl:350 =#
                let
                    #= logging.jl:351 =#
                    var"#104#level" = Base.CoreLogging.Info
                    #= logging.jl:352 =#
                    var"#105#std_level" = Base.CoreLogging.convert(Base.CoreLogging.LogLevel, var"#104#level")
                    #= logging.jl:353 =#
                    if var"#105#std_level" >= Base.CoreLogging._min_enabled_level[]
                        #= logging.jl:354 =#
                        var"#106#group" = :loading
                        #= logging.jl:355 =#
                        var"#107#_module" = Main
                        #= logging.jl:356 =#
                        var"#108#logger" = Base.CoreLogging.current_logger_for_env(var"#105#std_level", var"#106#group", var"#107#_module")
                        #= logging.jl:357 =#
                        if !(var"#108#logger" === Base.CoreLogging.nothing)
                            #= logging.jl:358 =#
                            var"#109#id" = :Main_75496674
                            #= logging.jl:361 =#
                            if Base.CoreLogging._invoked_shouldlog(var"#108#logger", var"#104#level", var"#107#_module", var"#106#group", var"#109#id")
                                #= logging.jl:362 =#
                                var"#110#file" = "/Users/anthony/.julia/packages/MLJModels/9mQfs/src/loading.jl"
                                #= logging.jl:363 =#
                                var"#111#line" = 168
                                #= logging.jl:364 =#
                                local var"#112#msg", var"#113#kwargs"
                                #= logging.jl:365 =#
                                begin
                                        #= logging.jl:308 =#
                                        var"#112#msg" = "For silent loading, specify `verbosity=0`. "
                                        #= logging.jl:309 =#
                                        var"#113#kwargs" = (;)
                                        #= logging.jl:310 =#
                                        true
                                    end && Base.CoreLogging.handle_message(var"#108#logger", var"#104#level", var"#112#msg", var"#107#_module", var"#106#group", var"#109#id", var"#110#file", var"#111#line"; var"#113#kwargs"...)
                            end
                        end
                    end
                    #= logging.jl:371 =#
                    Base.CoreLogging.nothing
                end
            end
        #= /Users/anthony/.julia/packages/MLJModels/9mQfs/src/loading.jl:169 =#
        var"#94#proxy" = (MLJModels.MLJModels).info("EvoTreeRegressor"; pkg = nothing, interactive = true)
        #= /Users/anthony/.julia/packages/MLJModels/9mQfs/src/loading.jl:172 =#
        var"#95#handle" = (MLJModels.MLJModels).Handle((var"#94#proxy").name, (var"#94#proxy").package_name)
        #= /Users/anthony/.julia/packages/MLJModels/9mQfs/src/loading.jl:173 =#
        var"#96#dic" = (MLJModels.MLJModels).INFO_GIVEN_HANDLE[var"#95#handle"]
        #= /Users/anthony/.julia/packages/MLJModels/9mQfs/src/loading.jl:174 =#
        (var"#97#path_str", var"#98#pkg_str") = (var"#96#dic"[:load_path], var"#96#dic"[:package_name])
        #= /Users/anthony/.julia/packages/MLJModels/9mQfs/src/loading.jl:175 =#
        var"#99#path" = var"#97#path_str" |> (MLJModels.Meta).parse
        #= /Users/anthony/.julia/packages/MLJModels/9mQfs/src/loading.jl:176 =#
        var"#100#pkg" = MLJModels.Symbol(var"#98#pkg_str")
    end
    begin
        #= /Users/anthony/.julia/packages/MLJModels/9mQfs/src/loading.jl:181 =#
        var"#101#path_components" = MLJModels.Symbol.(MLJModels.split(var"#97#path_str", '.'))                                                                                       
        #= /Users/anthony/.julia/packages/MLJModels/9mQfs/src/loading.jl:184 =#
        var"#102#api_pkg" = var"#101#path_components"[1]
        #= /Users/anthony/.julia/packages/MLJModels/9mQfs/src/loading.jl:185 =#
        var"#103#api_pkg_str" = MLJModels.string(var"#102#api_pkg")
        #= /Users/anthony/.julia/packages/MLJModels/9mQfs/src/loading.jl:187 =#
        if false || true
            #= /Users/anthony/.julia/packages/MLJModels/9mQfs/src/loading.jl:188 =#
            try
                #= /Users/anthony/.julia/packages/MLJModels/9mQfs/src/loading.jl:189 =#
                (MLJModels.MLJModels)._import(Main, var"#102#api_pkg", var"#100#pkg", false)
            catch
                #= /Users/anthony/.julia/packages/MLJModels/9mQfs/src/loading.jl:191 =#
                if true
                    #= /Users/anthony/.julia/packages/MLJModels/9mQfs/src/loading.jl:192 =#
                    (MLJModels.MLJModels).request("The package providing an interface " * "to `EvoTreeRegressor` " * "is not in your " * "current environment.\n" * "What do you want to do? ", "Install $(var"#103#api_pkg_str") in current environment.", "Abort.") == 1 || MLJModels.throw(MLJModels.InterruptException)
                end
                #= /Users/anthony/.julia/packages/MLJModels/9mQfs/src/loading.jl:202 =#
                (MLJModels.MLJModels)._import(Main, :Pkg, :Pkg, false)
                #= /Users/anthony/.julia/packages/MLJModels/9mQfs/src/loading.jl:203 =#
                (MLJModels.Pkg).add(var"#103#api_pkg_str")
            end
        end
        #= /Users/anthony/.julia/packages/MLJModels/9mQfs/src/loading.jl:206 =#
        (MLJModels.MLJModels)._import(Main, var"#102#api_pkg", var"#100#pkg", true)
        #= /Users/anthony/.julia/packages/MLJModels/9mQfs/src/loading.jl:207 =#
        (MLJModels.MLJModels)._eval(Main, var"#99#path")
    end
end

If I first manually add the required package, EvoTrees and then run @iload EvoTreeRegressor this issue disappears.

ablaom avatar May 26 '21 00:05 ablaom

julia> versioninfo()
Julia Version 1.6.0
Commit f9720dc2eb (2021-03-24 12:55 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin19.6.0)
  CPU: Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-11.0.1 (ORCJIT, skylake)
Environment:
  JULIA_LTS_PATH = /Applications/Julia-1.0.app/Contents/Resources/julia/bin/julia
  JULIA_PATH = /Applications/Julia-1.6.app/Contents/Resources/julia/bin/julia
  JULIA_EGLOT_PATH = /Applications/Julia-1.5.app/Contents/Resources/julia/bin/julia
  JULIA_NUM_THREADS = 5
  JULIA_NIGHTLY_PATH = /Applications/Julia-1.7.app/Contents/Resources/julia/bin/julia

ablaom avatar May 26 '21 00:05 ablaom

I tried also on macOS in case there is a weird platform-specific issue, but I still can't reproduce your problem there.

The expansion of the macro for you is the same as for me, apart from paths and gensym-generated variables. Not that I was expecting something different, but I figured it'd be good to double check it.

What about using Pkg.Types.Context().env.project.deps to check whether a package is in the current environment, instead of trying to use it and catching the error? The good thing is that it should work in all current versions of Julia since v1.3 (the first one currently supported), the drawback however is that you're hooking into Pkg internals (this is non-public API), so it may break in the future (but it can be dealt with with conditionals inside the macro and see below).

Also, note that Julia v1.7 has the feature of suggesting to install a package if not available, so the try/catch work you're doing here will be a duplicate already available in base Julia.

giordano avatar May 27 '21 13:05 giordano

@giordano Thanks for investigating.

What a pity you are unable to reproduce. I'm inclined to believe the problem lies outside of MLJModels. Why else would reproducibility be an issue? We are using identical Manifests/Project files, yes?

I will try your suggestion as a diagnostic. I'm not enthusiastic about it as a solution for the reasons you give.

ablaom avatar May 28 '21 02:05 ablaom