Distributions.jl
Distributions.jl copied to clipboard
JuliaStats
Quantile: use `Newton` method from `Roots.jl`
In https://github.com/JuliaStats/Distributions.jl/pull/1899, i'm trying to address the oscillation problem of Newton method, which does happen in real-world, non-synthetic distributions.
As per https://github.com/JuliaStats/Distributions.jl/pull/1899#issuecomment-2345526949, let's first stop reinventing the wheel.
Notably ] test now has:
┌ Warning: Circular dependency detected. Precompilation will be skipped for:
│ DistributionsTestExt [ffbe0ea5-a612-5ff7-aaf5-cac02eef3019]
│ DistributionsDensityInterfaceExt [8d3cd4d8-9481-53f7-8454-1bd38b07cca3]
│ DistributionsChainRulesCoreExt [6db1f127-056a-568b-bd49-ae61d42389fa]
└ @ Pkg.API /builddirs/julia/usr/share/julia/stdlib/v1.10/Pkg/src/API.jl:1279
Precompiling project...
1 dependency successfully precompiled in 5 seconds. 81 already precompiled. 3 skipped due to circular dependency.
1 dependency had output during precompilation:
┌ Distributions
│ WARNING: Distributions.MatrixReshaped is deprecated, use Distributions.ReshapedDistribution{2, S, D} where D<:Distributions.Distribution{Distributions.ArrayLikeVariate{1}, S} where S<:Distributions.ValueSupport instead.
│ likely near /repositories/Distributions.jl/src/deprecates.jl:58
│ WARNING: Distributions.MatrixReshaped is deprecated, use Distributions.ReshapedDistribution{2, S, D} where D<:Distributions.Distribution{Distributions.ArrayLikeVariate{1}, S} where S<:Distributions.ValueSupport instead.
│ likely near /repositories/Distributions.jl/src/deprecates.jl:58
│ WARNING: Distributions.MatrixReshaped is deprecated, use Distributions.ReshapedDistribution{2, S, D} where D<:Distributions.Distribution{Distributions.ArrayLikeVariate{1}, S} where S<:Distributions.ValueSupport instead.
│ likely near /repositories/Distributions.jl/src/deprecates.jl:58
│ WARNING: Distributions.MatrixReshaped is deprecated, use Distributions.ReshapedDistribution{2, S, D} where D<:Distributions.Distribution{Distributions.ArrayLikeVariate{1}, S} where S<:Distributions.ValueSupport instead.
│ likely near /repositories/Distributions.jl/src/deprecates.jl:58
│ WARNING: Distributions.MatrixReshaped is deprecated, use Distributions.ReshapedDistribution{2, S, D} where D<:Distributions.Distribution{Distributions.ArrayLikeVariate{1}, S} where S<:Distributions.ValueSupport instead.
│ likely near /repositories/Distributions.jl/src/deprecates.jl:58
└
Testing Running tests...
┌ Warning: Module DistributionsTestExt with build ID ffffffff-ffff-ffff-0000-220acb8184e8 is missing from the cache.
│ This may mean DistributionsTestExt [ffbe0ea5-a612-5ff7-aaf5-cac02eef3019] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:1948
┌ Error: Error during loading of extension DistributionsTestExt of Distributions, use `Base.retry_load_extensions()` to retry.
│ exception =
│ 1-element ExceptionStack:
│ Declaring __precompile__(false) is not allowed in files that are being precompiled.
│ Stacktrace:
│ [1] _require(pkg::Base.PkgId, env::Nothing)
│ @ Base ./loading.jl:1952
│ [2] __require_prelocked(uuidkey::Base.PkgId, env::Nothing)
│ @ Base ./loading.jl:1812
│ [3] #invoke_in_world#3
│ @ ./essentials.jl:926 [inlined]
│ [4] invoke_in_world
│ @ ./essentials.jl:923 [inlined]
│ [5] _require_prelocked
│ @ ./loading.jl:1803 [inlined]
│ [6] _require_prelocked
│ @ ./loading.jl:1802 [inlined]
│ [7] run_extension_callbacks(extid::Base.ExtensionId)
│ @ Base ./loading.jl:1295
│ [8] run_extension_callbacks(pkgid::Base.PkgId)
│ @ Base ./loading.jl:1330
│ [9] run_package_callbacks(modkey::Base.PkgId)
│ @ Base ./loading.jl:1164
│ [10] __require_prelocked(uuidkey::Base.PkgId, env::String)
│ @ Base ./loading.jl:1819
│ [11] #invoke_in_world#3
│ @ ./essentials.jl:926 [inlined]
│ [12] invoke_in_world
│ @ ./essentials.jl:923 [inlined]
│ [13] _require_prelocked(uuidkey::Base.PkgId, env::String)
│ @ Base ./loading.jl:1803
│ [14] macro expansion
│ @ ./loading.jl:1790 [inlined]
│ [15] macro expansion
│ @ ./lock.jl:267 [inlined]
│ [16] __require(into::Module, mod::Symbol)
│ @ Base ./loading.jl:1753
│ [17] #invoke_in_world#3
│ @ ./essentials.jl:926 [inlined]
│ [18] invoke_in_world
│ @ ./essentials.jl:923 [inlined]
│ [19] require(into::Module, mod::Symbol)
│ @ Base ./loading.jl:1746
│ [20] include
│ @ ./Base.jl:495 [inlined]
│ [21] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::String)
│ @ Base ./loading.jl:2222
│ [22] top-level scope
│ @ stdin:3
│ [23] eval
│ @ ./boot.jl:385 [inlined]
│ [24] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String)
│ @ Base ./loading.jl:2076
│ [25] include_string
│ @ ./loading.jl:2086 [inlined]
│ [26] exec_options(opts::Base.JLOptions)
│ @ Base ./client.jl:316
│ [27] _start()
│ @ Base ./client.jl:552
└ @ Base loading.jl:1301
Running tests:
Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:
Project coverage is 86.24%. Comparing base (
c2a7387) to head (f095427).
Additional details and impacted files
@@ Coverage Diff @@
## master #1900 +/- ##
==========================================
- Coverage 86.28% 86.24% -0.05%
==========================================
Files 146 146
Lines 8787 8775 -12
==========================================
- Hits 7582 7568 -14
- Misses 1205 1207 +2
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
:rocket: New features to boost your workflow:
- :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
And using also fails. I'm not sure i understand what the cycle is, so i'm not sure how to break it.
(@v1.10) pkg> activate --temp
Activating new project at `/tmp/jl_p7kNmk`
(jl_p7kNmk) pkg> add .
Updating git-repo `/repositories/Distributions.jl`
Resolving package versions...
Updating `/tmp/jl_p7kNmk/Project.toml`
[31c24e10] + Distributions v0.25.111 `../../repositories/Distributions.jl#roots.jl`
Updating `/tmp/jl_p7kNmk/Manifest.toml`
[7d9f7c33] + Accessors v0.1.37
[66dad0bd] + AliasTables v1.1.3
[d360d2e6] + ChainRulesCore v1.24.0
[38540f10] + CommonSolve v0.2.4
[34da2185] + Compat v4.16.0
[a33af91c] + CompositionsBase v0.1.2
[187b0558] + ConstructionBase v1.5.8
[9a962f9c] + DataAPI v1.16.0
[864edb3b] + DataStructures v0.18.20
[31c24e10] + Distributions v0.25.111 `../../repositories/Distributions.jl#roots.jl`
[ffbed154] + DocStringExtensions v0.9.3
[1a297f60] + FillArrays v1.13.0
[34004b35] + HypergeometricFunctions v0.3.24
[3587e190] + InverseFunctions v0.1.16
[92d709cd] + IrrationalConstants v0.2.2
[692b3bcd] + JLLWrappers v1.6.0
[2ab3a3ac] + LogExpFunctions v0.3.28
[1914dd2f] + MacroTools v0.5.13
[e1d29d7a] + Missings v1.2.0
[bac558e1] + OrderedCollections v1.6.3
[90014a1f] + PDMats v0.11.31
[21216c6a] + Preferences v1.4.3
[43287f4e] + PtrArrays v1.2.1
[1fd47b50] + QuadGK v2.11.0
[189a3867] + Reexport v1.2.2
[79098fc4] + Rmath v0.8.0
[f2b01f46] + Roots v2.1.8
[a2af1166] + SortingAlgorithms v1.2.1
[276daf66] + SpecialFunctions v2.4.0
[82ae8749] + StatsAPI v1.7.0
[2913bbd2] + StatsBase v0.34.3
[4c63d2b9] + StatsFuns v1.3.2
[efe28fd5] + OpenSpecFun_jll v0.5.5+0
[f50d1b31] + Rmath_jll v0.5.1+0
[0dad84c5] + ArgTools v1.1.1
[56f22d72] + Artifacts
[2a0f44e3] + Base64
[ade2ca70] + Dates
[f43a241f] + Downloads v1.6.0
[7b1f6079] + FileWatching
[b77e0a4c] + InteractiveUtils
[b27032c2] + LibCURL v0.6.4
[76f85450] + LibGit2
[8f399da3] + Libdl
[37e2e46d] + LinearAlgebra
[56ddb016] + Logging
[d6f4376e] + Markdown
[ca575930] + NetworkOptions v1.2.0
[44cfe95a] + Pkg v1.10.0
[de0858da] + Printf
[3fa0cd96] + REPL
[9a3f8284] + Random
[ea8e919c] + SHA v0.7.0
[9e88b42a] + Serialization
[6462fe0b] + Sockets
[2f01184e] + SparseArrays v1.10.0
[10745b16] + Statistics v1.10.0
[4607b0f0] + SuiteSparse
[fa267f1f] + TOML v1.0.3
[a4e569a6] + Tar v1.10.0
[8dfed614] + Test
[cf7118a7] + UUIDs
[4ec0a83e] + Unicode
[e66e0078] + CompilerSupportLibraries_jll v1.1.1+0
[deac9b47] + LibCURL_jll v8.4.0+0
[e37daf67] + LibGit2_jll v1.6.4+0
[29816b5a] + LibSSH2_jll v1.11.0+1
[c8ffd9c3] + MbedTLS_jll v2.28.2+1
[14a3606d] + MozillaCACerts_jll v2023.1.10
[4536629a] + OpenBLAS_jll v0.3.23+4
[05823500] + OpenLibm_jll v0.8.1+2
[bea87d4a] + SuiteSparse_jll v7.2.1+1
[83775a58] + Zlib_jll v1.2.13+1
[8e850b90] + libblastrampoline_jll v5.11.0+0
[8e850ede] + nghttp2_jll v1.52.0+1
[3f19e933] + p7zip_jll v17.4.0+2
┌ Warning: Circular dependency detected. Precompilation will be skipped for:
│ DistributionsTestExt [ffbe0ea5-a612-5ff7-aaf5-cac02eef3019]
│ DistributionsChainRulesCoreExt [6db1f127-056a-568b-bd49-ae61d42389fa]
└ @ Pkg.API /builddirs/julia/usr/share/julia/stdlib/v1.10/Pkg/src/API.jl:1279
julia> using Distributions
┌ Warning: Circular dependency detected. Precompilation will be skipped for:
│ DistributionsTestExt [ffbe0ea5-a612-5ff7-aaf5-cac02eef3019]
│ DistributionsChainRulesCoreExt [6db1f127-056a-568b-bd49-ae61d42389fa]
└ @ Pkg.API /builddirs/julia/usr/share/julia/stdlib/v1.10/Pkg/src/API.jl:1279
[ Info: Precompiling DistributionsTestExt [ffbe0ea5-a612-5ff7-aaf5-cac02eef3019]
┌ Warning: Module DistributionsTestExt with build ID ffffffff-ffff-ffff-0000-24fd0fec8fcc is missing from the cache.
│ This may mean DistributionsTestExt [ffbe0ea5-a612-5ff7-aaf5-cac02eef3019] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:1948
┌ Error: Error during loading of extension DistributionsTestExt of Distributions, use `Base.retry_load_extensions()` to retry.
│ exception =
│ 1-element ExceptionStack:
│ Declaring __precompile__(false) is not allowed in files that are being precompiled.
│ Stacktrace:
│ [1] _require(pkg::Base.PkgId, env::Nothing)
│ @ Base ./loading.jl:1952
│ [2] __require_prelocked(uuidkey::Base.PkgId, env::Nothing)
│ @ Base ./loading.jl:1812
│ [3] #invoke_in_world#3
│ @ ./essentials.jl:926 [inlined]
│ [4] invoke_in_world
│ @ ./essentials.jl:923 [inlined]
│ [5] _require_prelocked
│ @ ./loading.jl:1803 [inlined]
│ [6] _require_prelocked
│ @ ./loading.jl:1802 [inlined]
│ [7] run_extension_callbacks(extid::Base.ExtensionId)
│ @ Base ./loading.jl:1295
│ [8] run_extension_callbacks(pkgid::Base.PkgId)
│ @ Base ./loading.jl:1330
│ [9] run_package_callbacks(modkey::Base.PkgId)
│ @ Base ./loading.jl:1164
│ [10] __require_prelocked(uuidkey::Base.PkgId, env::String)
│ @ Base ./loading.jl:1819
│ [11] #invoke_in_world#3
│ @ ./essentials.jl:926 [inlined]
│ [12] invoke_in_world
│ @ ./essentials.jl:923 [inlined]
│ [13] _require_prelocked(uuidkey::Base.PkgId, env::String)
│ @ Base ./loading.jl:1803
│ [14] macro expansion
│ @ ./loading.jl:1790 [inlined]
│ [15] macro expansion
│ @ ./lock.jl:267 [inlined]
│ [16] __require(into::Module, mod::Symbol)
│ @ Base ./loading.jl:1753
│ [17] #invoke_in_world#3
│ @ ./essentials.jl:926 [inlined]
│ [18] invoke_in_world
│ @ ./essentials.jl:923 [inlined]
│ [19] require(into::Module, mod::Symbol)
│ @ Base ./loading.jl:1746
│ [20] include
│ @ ./Base.jl:495 [inlined]
│ [21] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::String)
│ @ Base ./loading.jl:2222
│ [22] top-level scope
│ @ stdin:3
│ [23] eval
│ @ ./boot.jl:385 [inlined]
│ [24] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String)
│ @ Base ./loading.jl:2076
│ [25] include_string
│ @ ./loading.jl:2086 [inlined]
│ [26] exec_options(opts::Base.JLOptions)
│ @ Base ./client.jl:316
│ [27] _start()
│ @ Base ./client.jl:552
└ @ Base loading.jl:1301
(jl_p7kNmk) pkg> precompile
┌ Warning: Circular dependency detected. Precompilation will be skipped for:
│ DistributionsTestExt [ffbe0ea5-a612-5ff7-aaf5-cac02eef3019]
│ DistributionsChainRulesCoreExt [6db1f127-056a-568b-bd49-ae61d42389fa]
└ @ Pkg.API /builddirs/julia/usr/share/julia/stdlib/v1.10/Pkg/src/API.jl:1279
julia> Base.retry_load_extensions()
julia>
The problem is
(jl_PRakDJ) pkg> why Test
Roots → Accessors → Test
(jl_PRakDJ) pkg> why ChainRulesCore
Roots → ChainRulesCore
I made a PR to Roots that would fix the ChainRulesCore issue: https://github.com/JuliaMath/Roots.jl/pull/445
Next issue:
https://github.com/JuliaStats/Distributions.jl/actions/runs/10855139692/job/30127082809?pr=1900#step:6:664
https://github.com/JuliaStats/Distributions.jl/blob/b219803a0d03a7c75d7aef7c0bab6cd0d79997dc/test/multivariate/dirichlet.jl#L137-L142
test_frule: Dirichlet on Vector{Float64}: Error During Test at /home/runner/.julia/packages/ChainRulesTestUtils/Ko1Wr/src/testers.jl:123
Got exception outside of a @test
DomainError with [0.100746, 0.151104, 0.198606, 0.106677, 0.417981, 0.234742, 0.910606, 0.501805, 0.861964, -0.00290635]:
Dirichlet: alpha must be a positive vector.
Stacktrace:
[1] #505
@ ~/work/Distributions.jl/Distributions.jl/src/multivariate/dirichlet.jl:29 [inlined]
[2] check_args
@ ~/work/Distributions.jl/Distributions.jl/src/utils.jl:89 [inlined]
[3] (Dirichlet{Float64, Ts, S} where {Ts<:AbstractVector{Float64}, S<:Real})(alpha::Vector{Float64}; check_args::Bool)
@ Distributions ~/work/Distributions.jl/Distributions.jl/src/multivariate/dirichlet.jl:29
[4] #Dirichlet#510
@ ~/work/Distributions.jl/Distributions.jl/src/multivariate/dirichlet.jl:40 [inlined]
[5] Dirichlet(alpha::Vector{Float64})
@ Distributions ~/work/Distributions.jl/Distributions.jl/src/multivariate/dirichlet.jl:40
[6] (::ChainRulesTestUtils.var"#call_on_copy#61"{NamedTuple{(), Tuple{}}})(::Any, ::Any, ::Vararg{Any, N} where N)
@ ChainRulesTestUtils ~/.julia/packages/ChainRulesTestUtils/Ko1Wr/src/testers.jl:121
[7] (::ChainRulesTestUtils.var"#fnew#57"{ChainRulesTestUtils.var"#call_on_copy#61"{NamedTuple{(), Tuple{}}}, Tuple{UnionAll, Vector{Float64}}, Tuple{Bool, Bool}})(sigargs::Any)
@ ChainRulesTestUtils ~/.julia/packages/ChainRulesTestUtils/Ko1Wr/src/finite_difference_calls.jl:86
[8] (::FiniteDifferences.var"#88#89"{ChainRulesTestUtils.var"#fnew#57"{ChainRulesTestUtils.var"#call_on_copy#61"{NamedTuple{(), Tuple{}}}, Tuple{UnionAll, Vector{Float64}}, Tuple{Bool, Bool}}, typeof(identity)})(x_vec::Vector{Float64})
@ FiniteDifferences ~/.julia/packages/FiniteDifferences/IPGFN/src/grad.jl:60
[9] (::FiniteDifferences.var"#86#87"{FiniteDifferences.var"#88#89"{ChainRulesTestUtils.var"#fnew#57"{ChainRulesTestUtils.var"#call_on_copy#61"{NamedTuple{(), Tuple{}}}, Tuple{UnionAll, Vector{Float64}}, Tuple{Bool, Bool}}, typeof(identity)}, Vector{Float64}, Vector{Float64}})(ε::Float64)
@ FiniteDifferences ~/.julia/packages/FiniteDifferences/IPGFN/src/grad.jl:48
[10] newf
@ ~/.julia/packages/StaticArrays/MSJcA/src/broadcast.jl:186 [inlined]
[11] macro expansion
@ ~/.julia/packages/StaticArrays/MSJcA/src/broadcast.jl:135 [inlined]
[12] __broadcast
@ ~/.julia/packages/StaticArrays/MSJcA/src/broadcast.jl:123 [inlined]
[13] _broadcast
@ ~/.julia/packages/StaticArrays/MSJcA/src/broadcast.jl:119 [inlined]
[14] copy
@ ~/.julia/packages/StaticArrays/MSJcA/src/broadcast.jl:60 [inlined]
[15] materialize
@ ./broadcast.jl:883 [inlined]
[16] _eval_function(m::FiniteDifferences.UnadaptedFiniteDifferenceMethod{7, 5}, f::FiniteDifferences.var"#86#87"{FiniteDifferences.var"#88#89"{ChainRulesTestUtils.var"#fnew#57"{ChainRulesTestUtils.var"#call_on_copy#61"{NamedTuple{(), Tuple{}}}, Tuple{UnionAll, Vector{Float64}}, Tuple{Bool, Bool}}, typeof(identity)}, Vector{Float64}, Vector{Float64}}, x::Float64, step::Float64)
@ FiniteDifferences ~/.julia/packages/FiniteDifferences/IPGFN/src/methods.jl:249
[17] _estimate_magnitudes(m::FiniteDifferences.UnadaptedFiniteDifferenceMethod{7, 5}, f::FiniteDifferences.var"#86#87"{FiniteDifferences.var"#88#89"{ChainRulesTestUtils.var"#fnew#57"{ChainRulesTestUtils.var"#call_on_copy#61"{NamedTuple{(), Tuple{}}}, Tuple{UnionAll, Vector{Float64}}, Tuple{Bool, Bool}}, typeof(identity)}, Vector{Float64}, Vector{Float64}}, x::Float64)
@ FiniteDifferences ~/.julia/packages/FiniteDifferences/IPGFN/src/methods.jl:378
[18] estimate_step(m::FiniteDifferences.AdaptedFiniteDifferenceMethod{5, 1, FiniteDifferences.UnadaptedFiniteDifferenceMethod{7, 5}}, f::FiniteDifferences.var"#86#87"{FiniteDifferences.var"#88#89"{ChainRulesTestUtils.var"#fnew#57"{ChainRulesTestUtils.var"#call_on_copy#61"{NamedTuple{(), Tuple{}}}, Tuple{UnionAll, Vector{Float64}}, Tuple{Bool, Bool}}, typeof(identity)}, Vector{Float64}, Vector{Float64}}, x::Float64)
@ FiniteDifferences ~/.julia/packages/FiniteDifferences/IPGFN/src/methods.jl:365
[19] (::FiniteDifferences.AdaptedFiniteDifferenceMethod{5, 1, FiniteDifferences.UnadaptedFiniteDifferenceMethod{7, 5}})(f::FiniteDifferences.var"#86#87"{FiniteDifferences.var"#88#89"{ChainRulesTestUtils.var"#fnew#57"{ChainRulesTestUtils.var"#call_on_copy#61"{NamedTuple{(), Tuple{}}}, Tuple{UnionAll, Vector{Float64}}, Tuple{Bool, Bool}}, typeof(identity)}, Vector{Float64}, Vector{Float64}}, x::Float64)
@ FiniteDifferences ~/.julia/packages/FiniteDifferences/IPGFN/src/methods.jl:193
[20] _jvp
@ ~/.julia/packages/FiniteDifferences/IPGFN/src/grad.jl:48 [inlined]
[21] jvp(fdm::FiniteDifferences.AdaptedFiniteDifferenceMethod{5, 1, FiniteDifferences.UnadaptedFiniteDifferenceMethod{7, 5}}, f::ChainRulesTestUtils.var"#fnew#57"{ChainRulesTestUtils.var"#call_on_copy#61"{NamedTuple{(), Tuple{}}}, Tuple{UnionAll, Vector{Float64}}, Tuple{Bool, Bool}}, ::Tuple{Vector{Float64}, Vector{Float64}})
@ FiniteDifferences ~/.julia/packages/FiniteDifferences/IPGFN/src/grad.jl:60
[22] _make_jvp_call(fdm::Any, f::Any, y::Any, xs::Any, ẋs::Any, ignores::Any)
@ ChainRulesTestUtils ~/.julia/packages/ChainRulesTestUtils/Ko1Wr/src/finite_difference_calls.jl:24
[23] macro expansion
@ ~/.julia/packages/ChainRulesTestUtils/Ko1Wr/src/testers.jl:142 [inlined]
[24] macro expansion
@ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1151 [inlined]
[25] test_frule(::RuleConfig, ::Any, ::Any, ::Vararg{Any, N} where N; output_tangent::Any, fdm::Any, frule_f::Any, check_inferred::Bool, fkwargs::NamedTuple, rtol::Real, atol::Real, testset_name::Any, kwargs::Any)
@ ChainRulesTestUtils ~/.julia/packages/ChainRulesTestUtils/Ko1Wr/src/testers.jl:125
[26] test_frule(::Any, ::Vararg{Any, N} where N; kwargs::Any)
@ ChainRulesTestUtils ~/.julia/packages/ChainRulesTestUtils/Ko1Wr/src/testers.jl:100
[27] macro expansion
@ ~/work/Distributions.jl/Distributions.jl/test/multivariate/dirichlet.jl:140 [inlined]
[28] macro expansion
@ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1226 [inlined]
[29] macro expansion
@ ~/work/Distributions.jl/Distributions.jl/test/multivariate/dirichlet.jl:137 [inlined]
[30] top-level scope
@ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1226
[31] include
@ ./client.jl:444 [inlined]
[32] macro expansion
@ ~/work/Distributions.jl/Distributions.jl/test/runtests.jl:175 [inlined]
[33] macro expansion
@ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1226 [inlined]
[34] macro expansion
@ ~/work/Distributions.jl/Distributions.jl/test/runtests.jl:174 [inlined]
[35] macro expansion
@ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1151 [inlined]
[36] top-level scope
@ ~/work/Distributions.jl/Distributions.jl/test/runtests.jl:174
[37] include(fname::String)
@ Base.MainInclude ./client.jl:444
[38] top-level scope
@ none:6
[39] eval
@ ./boot.jl:360 [inlined]
[40] exec_options(opts::Base.JLOptions)
@ Base ./client.jl:261
[41] _start()
@ Base ./client.jl:485
I suspect it was always there and just not caught by CI because only two jula versions are being tested...
Looking at the wider picture,
- aqua ambiguity test fails until julia 1.9
- julia-1.7 based CI seems to be fully green
- julia-1.8 based CI fails on macos: ~56
DomainErrorfailures. I don't know what to make of them. (log will only return a complex result if called with a complex argument. Try log(Complex(x)).) - julia-1.6 based CI seems to be fully red (when using macos-13 instead of macos-14) (
Dirichlet: alpha must be a positive vector.)
All of these happen in the current master regardless of this PR.
aqua ambiguity test fails until julia 1.9
It is somewhat expected that this highly depends on the Julia versions as some of the ambiguities might disappear due to upstream changes which might not be available in older Julia or package versions (the latest package versions might not be installable on older Julia versions). The last time I checked ambiguity issues were caused by upstream packages, so there's nothing we can do about it (and if users are actually affected by them, they can always update to a newer Julia versions).
julia-1.8 based CI fails on macos: ~56 DomainError failures. I don't know what to make of them. (log will only return a complex result if called with a complex argument. Try log(Complex(x)).)
julia-1.6 based CI seems to be fully red (when using macos-13 instead of macos-14) (Dirichlet: alpha must be a positive vector.)
I assume the 1.8 errors are also caused by AD tests? I think there's nothing to be worried about these (apart from that it causes CI failures) since such errors often show up when performing AD tests close to the boundary of the domain (perturbations might lead to e.g. negative values or negative values of which we compute log), in particular if they are not fine-tuned or if the RNG (seed) changes (as it can happen on different Julia versions).
If these are the only test failures on older Julia versions, I actually think there are no major problems with these Julia versions.
aqua ambiguity test fails until julia 1.9
It is somewhat expected that this highly depends on the Julia versions as some of the ambiguities might disappear due to upstream changes which might not be available in older Julia or package versions (the latest package versions might not be installable on older Julia versions). The last time I checked ambiguity issues were caused by upstream packages, so there's nothing we can do about it (and if users are actually affected by them, they can always update to a newer Julia versions).
julia-1.8 based CI fails on macos: ~56 DomainError failures. I don't know what to make of them. (log will only return a complex result if called with a complex argument. Try log(Complex(x)).)
julia-1.6 based CI seems to be fully red (when using macos-13 instead of macos-14) (Dirichlet: alpha must be a positive vector.)
I assume the 1.8 errors are also caused by AD tests?
I can't tell: https://github.com/LebedevRI/Distributions.jl/actions/runs/10910190269/job/30280113268#step:6:1037 It's weird that it only fails on macos.
I think there's nothing to be worried about these (apart from that it causes CI failures) since such errors often show up when performing AD tests close to the boundary of the domain (perturbations might lead to e.g. negative values or negative values of which we compute
log), in particular if they are not fine-tuned or if the RNG (seed) changes (as it can happen on different Julia versions).If these are the only test failures on older Julia versions, I actually think there are no major problems with these Julia versions.
Yes, but what about the CI? Should CI test j1.7 instead of (failing) j1.6? Or should the failing tests be simply disabled for bad julia version?
I've looked, and this randomness randomness is a rather known issue: https://docs.julialang.org/en/v1/stdlib/Random/#Reproducibility I suspect the best outcome is to use https://github.com/JuliaRandom/StableRNGs.jl, but 1. that results in +40% slower total test time, 2. does not seem to help, i'm not sure why.
All in all, i've spent more time than i would have wanted, and until there's actionable feedback, i've just gone ahead and skipped julia 1.6 in CI to 1.7, while keeping minimal version still at 1.6.
I'm not sure if this is waiting on something from my side, in which case that needs to be communicated more clearly.
@devmotion Hi! Now that julia-1.10 is required anyways, i believe this PR became trivial.