timholy
Fix struct/const revision
The aim here is to provide full support for struct/const revision. This first commit just adds simple tests, but it already reveals something that needs to be addressed: when a struct gets revised, methods defined for that struct aren't automatically re-evaluated for the new type. Specifically, in the added tests, after revising the definition of Point we get
julia> methods(StructConst.firstval)
# 1 method for generic function "firstval" from StructConst:
[1] firstval(p::@world(StructConst.Point, 40598:40739))
@ /tmp/2wfLvVmwY7/StructConst/src/StructConst.jl:11
and there is no defined method for a currently-valid StructConst.Point.
@Keno, if you have a moment, let me check my understanding of the current situation:
- method applicability is assessed by type-intersection, as usual, but now types (via their bindings) can have a world-age
- the MethodInstances and CodeInstances involving the old type are still valid (their world age is not capped, but of course they require inputs with old types)
- there is no invalidation step needed for changes of the kind tested with the revision of Point (
invalidate_code_for_globalref!is not called?) - we can't count on
binding.backedgesto list everything that uses this binding - the right solution is for Revise to:
- detect that we're about to redefine a struct
- search the entire system for any methods where that type appears in a signature
- parse the source files, if necessary
- redefine the struct
- re-evaluate the method definitions
Presumably, waiting to do the last step as the last stage of a revision would be wise, as it is possible that more than one struct will be revised at the same time, and one might as well do each evaluation only once.
- method applicability is assessed by type-intersection, as usual, but now types (via their bindings) can have a world-age
I think that's a valid way to say it, but just to clarify, nothing about the type object itself changes, just the name by which you have to access it
- the MethodInstances and CodeInstances involving the old type are still valid (their world age is not capped, but of course they require inputs with old types)
yes
- there is no invalidation step needed for changes of the kind tested with the revision of Point (
invalidate_code_for_globalref!is not called?)
I don't understand what you're asking
- we can't count on
binding.backedgesto list everything that uses this binding
It lists all cross-module edges to get everything, you also need to iterate all methods in the same module as the binding.
the right solution is for Revise to:
- detect that we're about to redefine a struct
- search the entire system for any methods where that type appears in a signature
- parse the source files, if necessary
- redefine the struct
- re-evaluate the method definitions
In the fullness of time, the correct way to implement Revise is to run the toplevel expressions in a non-standard evaluation mode that tracks dependency edges for any evaluated constants. However, that should maybe wait until after the JuliaLowering changes and the interpreter rewrite. I think your proposed strategy is reasonable in the meantime although not fully complete.
there is no invalidation step needed for changes of the kind tested with the revision of Point (
invalidate_code_for_globalref!is not called?)I don't understand what you're asking
I was wondering whether some kind of callback in there could be helpful in short-circuiting the process of determining which methoddefs need to be re-evaluated, but if it's not called then that won't help. From what I can tell, that's not going to work anyway because it only traverses the module containing the type definition. But your comment
It lists all cross-module edges to get everything, you also need to iterate all methods in the same module as the binding.
seems to be what I was looking for.
This seems fairly straightforward, thanks for the excellent groundwork.
I was wondering whether some kind of callback in there could be helpful in short-circuiting the process of determining which methoddefs need to be re-evaluated
No, there's no edges of any kind for signatures. You need to scan the entire system. However, an earlier version of that code did the same whole-system scan, so you can steal some code for whole-system scanning.
I added a cross-module test (the StructConstUser module), expecting this might populate b.bindings, but I get
julia> b = convert(Core.Binding, GlobalRef(StructConst, :Point))
Binding StructConst.Point
40743:∞ - constant binding to StructConst.Point
40598:40742 - constant binding to @world(StructConst.Point, 40598:40742)
1:0 - backdated constant binding to @world(StructConst.Point, 40598:40742)
1:0 - backdated constant binding to @world(StructConst.Point, 40598:40742)
julia> b.backedges
ERROR: UndefRefError: access to undefined reference
Stacktrace:
[1] getproperty(x::Core.Binding, f::Symbol)
@ Base ./Base_compiler.jl:55
[2] top-level scope
@ REPL[6]:1
so I don't think I understand what
It lists all cross-module edges
really means. It also doesn't populate if I add
struct PointWrapper
p::StructConst.Point
end
to that module.
I understand that I have to traverse the whole system, I'm just curious about what b.backedges stores.
There's two kinds of edges that are tracked there. One is explicit import/using. The other is to lowered code of method definitions (but only after the first inference). At no point is an edge ever added for an evaluation of a binding, only for compilation of code that references the binding.
Just to cross-check:
module StructConstUser
using StructConst: Fixed, Point
struct PointWrapper
p::Point
end
scuf(f::Fixed) = 33 * f.x
scup(p::Point) = 44 * p.x
end
yields a backedge for the using StructConst: Fixed, Point statement, but I don't see anything related to scup even if I evaluate it:
julia> e = only(b.backedges)
Binding StructConstUser.Point
40598:∞ - explicit `using` from StructConst.Point
1:0 - undefined binding - guard entry
but I don't see anything related to
scupeven if I evaluate it:
That's because scup doesn't use any bindings other than * and getproperty in its lowered code (as I said, no edges for signatures). However, even if you had like mkscup() = Point(), that binding would be to StructConstUser. To actually get a cross-module edge is not that easy, but you could do @eval mkscup() = $(GlobalRef(StructConst, :Point))(). The most common case to get cross-module edges is macros. Also, not a lot of edges is good, that is the design goal, since we don't want to store them :).
This is getting close, but there's a philosophical question to answer: should Revise try to hew to Base behavior as closely as possible, or should it add/modify behaviors for what might be a nicer interactive experience?
In this case, the issue is the following: when I redefine Point, in Base you can still call previously-defined functions on "old data." However, in this PR, calling functions on old data throws (or is intended to throw) a MethodError. Why might you want that? Because I imagine that a common user-error will be to forget to refresh your data with updated types, and if your whole code pipeline runs (using all the old definitions), I imagine people will bang their heads against walls trying to figure out why their edits aren't having the intended effect. In contrast, if they get a no method foo(::@world(Point, m:n)), they'll pretty quickly learn how you fix it.
I should say that initially I wasn't aiming in this direction, and this current proposal arose from noticing some unexpected subtleties about the order in which Revise's cache files get set up: the order in which you parse the source, lower the code, and rebind the name matters quite a lot, and some of the choices in this version are designed to compensate for the fact that Revise normally doesn't parse a file unless it's been edited. (Now, we'll also have to parse any file with a method whose signature references an updated type.) I think it's possible to hew to the Base behavior, if we decide that's better, but the initial bugs I faced led me to ask what behavior we actually want, and I came to the conclusion that it's likely a better user experience if we invalidate methods that only work on old types.
However, in this PR, calling functions on old data throws (or is intended to throw) a
MethodError. Why might you want that?
I think that's fine. Conceptually Revise does both adding new methods and deleting old ones. Base is always "append only".
The collateral damage to tests is something I'm looking into; if one runs all the tests, there appear to be some failures that predate this effort, including an incomplete updating of the ecosystem to https://github.com/JuliaLang/julia/pull/52415. I'll work on that.
But meanwhile, this seems to be working:
tim@diva:~/.julia/dev/Revise$ ~/src/juliaw/julia --startup=no --project -e 'using Pkg; Pkg.test(; test_args=["struct/const revision"], coverage=false)'
Testing Revise
Status `/tmp/jl_4EbO8B/Project.toml`
# Pkg output deleted
Testing Running tests...
Test Summary: | Pass Total Time
Revise | 30 30 10.3s
beginning cleanup
and may be ready for anyone who wants to review it.
@Keno @vtjnash @aviatesk @JeffBezanson here's a fun one:
julia> using Revise
Precompiling Revise finished.
1 dependency successfully precompiled in 6 seconds. 19 already precompiled.
julia> Revise.track(Core.Compiler)
julia> fieldnames(Core.Compiler.NativeInterpreter)
(:world, :method_table, :inf_cache, :codegen, :inf_params, :opt_params)
#
# edited Compiler/src/types.jl to add a field `dummy` to `NativeInterpreter`
#
julia> 1+1
WARNING: Detected access to binding `Compiler.#NativeInterpreter#476` in a world prior to its definition world.
Julia 1.12 has introduced more strict world age semantics for global bindings.
!!! This code may malfunction under Revise.
!!! This code will error in future versions of Julia.
Hint: Add an appropriate `invokelatest` around the access to this binding.
⋮ # lots more like this
Compiling the compiler. This may take several minutes ...
Base.Compiler ──── 1.72376 seconds
2
julia> fieldnames(Core.Compiler.NativeInterpreter)
(:world, :method_table, :inf_cache, :codegen, :inf_params, :opt_params, :dummy)
julia> function f(::Integer)
Base.Experimental.@force_compile
return 1
end
f (generic function with 1 method)
julia> f(5)
1
This is interesting.
So it looks like this PR tracks edges not only from binding to method signature, but also from binding to method body?
I think a similar mechanism will be needed when developing a new language server. If it's already implemented in Revise, I'd like to reuse it.
I haven't had a chance to look at this PR closely yet, but I'll read it later.
Yes, when a type gets rebound, Revise will scan the entire system for methods m::Method where that type appears in m.sig. It then tries to find the expression that defined that method, deletes the old method, and re-evaluates the expression. That way it regenerates methods for the new meaning of the type.
There's a separate bit of novelty for Core.Compiler: this PR also checks whether any of the entry points to type inference have been invalidated, and if so calls bootstrap!
I'll be curious to see whether it improves the efficiency of hacking on inference.
There's a separate bit of novelty for
Core.Compiler: this PR also checks whether any of the entry points to type inference have been invalidated, and if so callsbootstrap!
I don't know if I like that. We already have @activate Compiler[:codegen] for this use case.
I didn't know about @activate. Is there a demo of the workflow somewhere? I've seen the docs, and they're good, but I suspect there's a bit more wisdom around effective usage than they convey.
I can back that commit out, if there are other mechanisms to achieve the same aim. Are you saying that Revise.track(Core.Compiler) should be deprecated?
I tried this. On:
- Revise (this PR)
- LoweredCodeUtils master
- JuliaInterpreter master
- Julia 1.12.0-beta1
What I did was:
- Load Revise, Tensors.
- Make the following revision to Tensors:
diff --git a/src/Tensors.jl b/src/Tensors.jl index 4cd677b..5f04312 100644 --- a/src/Tensors.jl +++ b/src/Tensors.jl @@ -63,6 +63,7 @@ julia> Tensor{1,3,Float64}((1.0, 2.0, 3.0)) """ struct Tensor{order, dim, T, M} <: AbstractTensor{order, dim, T} data::NTuple{M, T} + x::Int Tensor{order, dim, T, M}(data::NTuple) where {order, dim, T, M} = new{order, dim, T, M}(data) end - Press 1 in the REPL
- Press 1 in the REPL again
with the following result:
julia> using Revise, Tensors
# updates Tensors file here
julia> 1
ERROR: UndefVarError: `Broadcasted` not defined in `StaticArrays.StableFlatten`
Suggestion: check for spelling errors or missing imports.
Stacktrace:
[1] top-level scope
@ ~/.julia/packages/StaticArrays/LSPcF/src/broadcast.jl:180
Revise evaluation error at /Users/kristoffercarlsson/.julia/packages/StaticArrays/LSPcF/src/broadcast.jl:180
Stacktrace:
[1] methods_by_execution!(recurse::Any, methodinfo::Revise.CodeTrackingMethodInfo, docexprs::Dict{…}, mod::Module, ex::Expr; mode::Symbol, disablebp::Bool, always_rethrow::Bool, kwargs::@Kwargs{})
@ Revise ~/JuliaPkgs/Revise.jl/src/lowered.jl:306
[2] eval_with_signatures(mod::Module, ex::Expr; mode::Symbol, kwargs::@Kwargs{})
@ Revise ~/JuliaPkgs/Revise.jl/src/packagedef.jl:556
[3] eval_with_signatures
@ ~/JuliaPkgs/Revise.jl/src/packagedef.jl:553 [inlined]
[4] instantiate_sigs!(modexsigs::OrderedCollections.OrderedDict{…}; mode::Symbol, kwargs::@Kwargs{})
@ Revise ~/JuliaPkgs/Revise.jl/src/packagedef.jl:564
[5] instantiate_sigs!
@ ~/JuliaPkgs/Revise.jl/src/packagedef.jl:560 [inlined]
[6] maybe_extract_sigs_for_meths(meths::Set{Method})
@ Revise ~/JuliaPkgs/Revise.jl/src/pkgs.jl:162
[7] (::Revise.var"#107#108"{Bool})()
@ Revise ~/JuliaPkgs/Revise.jl/src/packagedef.jl:941
[8] lock(f::Revise.var"#107#108"{Bool}, l::ReentrantLock)
@ Base ./lock.jl:335
[9] #revise#104
@ ~/JuliaPkgs/Revise.jl/src/packagedef.jl:844 [inlined]
[10] revise()
@ Revise ~/JuliaPkgs/Revise.jl/src/packagedef.jl:842
[11] top-level scope
@ REPL:1
caused by: UndefVarError: `Broadcasted` not defined in `StaticArrays.StableFlatten`
Suggestion: check for spelling errors or missing imports.
Stacktrace:
[1] lookup_var
@ ~/.julia/packages/JuliaInterpreter/J7J9G/src/interpret.jl:5 [inlined]
[2] step_expr!(recurse::Any, frame::JuliaInterpreter.Frame, node::Any, istoplevel::Bool)
@ JuliaInterpreter ~/.julia/packages/JuliaInterpreter/J7J9G/src/interpret.jl:44
[3] signature(recurse::Any, frame::JuliaInterpreter.Frame, stmt::Any, pc::Int64)
@ LoweredCodeUtils ~/.julia/packages/LoweredCodeUtils/BIVzf/src/signatures.jl:50
[4] methoddef!(recurse::Any, signatures::Vector{Any}, frame::JuliaInterpreter.Frame, stmt::Any, pc::Int64; define::Bool)
@ LoweredCodeUtils ~/.julia/packages/LoweredCodeUtils/BIVzf/src/signatures.jl:595
[5] methoddef!
@ ~/.julia/packages/LoweredCodeUtils/BIVzf/src/signatures.jl:534 [inlined]
[6] methods_by_execution!(recurse::Any, methodinfo::Revise.CodeTrackingMethodInfo, docexprs::Dict{…}, frame::JuliaInterpreter.Frame, isrequired::Vector{…}; mode::Symbol, skip_include::Bool)
@ Revise ~/JuliaPkgs/Revise.jl/src/lowered.jl:354
[7] methods_by_execution!
@ ~/JuliaPkgs/Revise.jl/src/lowered.jl:317 [inlined]
[8] methods_by_execution!(recurse::Any, methodinfo::Revise.CodeTrackingMethodInfo, docexprs::Dict{…}, mod::Module, ex::Expr; mode::Symbol, disablebp::Bool, always_rethrow::Bool, kwargs::@Kwargs{})
@ Revise ~/JuliaPkgs/Revise.jl/src/lowered.jl:296
[9] eval_with_signatures(mod::Module, ex::Expr; mode::Symbol, kwargs::@Kwargs{})
@ Revise ~/JuliaPkgs/Revise.jl/src/packagedef.jl:556
[10] eval_with_signatures
@ ~/JuliaPkgs/Revise.jl/src/packagedef.jl:553 [inlined]
[11] instantiate_sigs!(modexsigs::OrderedCollections.OrderedDict{…}; mode::Symbol, kwargs::@Kwargs{})
@ Revise ~/JuliaPkgs/Revise.jl/src/packagedef.jl:564
[12] instantiate_sigs!
@ ~/JuliaPkgs/Revise.jl/src/packagedef.jl:560 [inlined]
[13] maybe_extract_sigs_for_meths(meths::Set{Method})
@ Revise ~/JuliaPkgs/Revise.jl/src/pkgs.jl:162
[14] (::Revise.var"#107#108"{Bool})()
@ Revise ~/JuliaPkgs/Revise.jl/src/packagedef.jl:941
[15] lock(f::Revise.var"#107#108"{Bool}, l::ReentrantLock)
@ Base ./lock.jl:335
[16] #revise#104
@ ~/JuliaPkgs/Revise.jl/src/packagedef.jl:844 [inlined]
[17] revise()
@ Revise ~/JuliaPkgs/Revise.jl/src/packagedef.jl:842
[18] top-level scope
@ REPL:1
Some type information was truncated. Use `show(err)` to see complete types.
julia> 1
Compiling the compiler. This may take several minutes ...
Base.Compiler ──── 3.43502 seconds
1
The UndefVarError seems correct:
julia> Tensors.StaticArrays.StableFlatten.Broadcasted
ERROR: UndefVarError: `Broadcasted` not defined in `StaticArrays.StableFlatten`
Suggestion: check for spelling errors or missing imports.
Stacktrace:
[1] getproperty(x::Module, f::Symbol)
@ Base ./Base_compiler.jl:48
[2] top-level scope
@ REPL[4]:1
so I guess the question is why it is tried to be looked up. And also, why does the Compiler recompile when I press another time 1 with no new modifications?
I updated against the latest master branch.
I tested this branch with the simple case, but it seems that for exported bindings, additional edge tracking is needed?
In particular, the world ages of bindings that are usinged need to be updated, it seems.
For example, let's say we have an Example package like this:
module Example
export hello, Hello
struct Hello
who::String
end
hello(x::Hello) = hello(x.who)
"""
hello(who::String)
Return "Hello, `who`".
"""
hello(who::String) = "Hello, $who"
end
We get the following error:
julia> using Revise
julia> using Example
julia> hello(Hello("world"))
"Hello, world"
julia> # Apply the following diff
# ```diff
# diff --git a/src/Example.jl b/src/Example.jl
# index 65c7eae..c631814 100644
# --- a/src/Example.jl
# +++ b/src/Example.jl
# @@ -2,10 +2,10 @@ module Example
# export hello, Hello
#
# struct Hello
# - who::String
# + who2::String
# end
#
# -hello(x::Hello) = hello(x.who)
# +hello(x::Hello) = hello(x.who2 * " (changed)")
#
# """
# hello(who::String)
# ```
julia> hello(Hello("world"))
ERROR: MethodError: no method matching @world(Example.Hello, 38355:38383)(::String)
The type `@world(Example.Hello, 38355:38383)` exists, but no method is defined for this combination of argument types when trying to construct it.
Stacktrace:
[1] top-level scope
@ REPL[5]:1
julia> hello(Example.Hello("world")) # but this works
"Hello, world (changed)"
@KristofferC your example looks like a selective evaluation issue: https://github.com/JuliaArrays/StaticArrays.jl/blob/e7c5e0162fe829f317b1d49aea7f192eb8c58eae/src/broadcast.jl#L171-L175 doesn't run because of the Julia version number, but somehow Revise still tries to step through it.
I've found that the problematic frame looks like this:
julia> Revise.LoweredCodeUtils.print_with_code(stdout, frame.framecode.src, isrequired)
1 t 1 ─ %1 = StaticArrays.StableFlatten.:>=
2 t │ %2 = StaticArrays.StableFlatten.VERSION
3 t │ %3 = dynamic (%1)(%2, v"1.11.0-DEV.103")
4 t └── goto #3 if not %3
5 t 2 ─ %5 = StaticArrays.StableFlatten.Broadcast
6 t │ %6 = dynamic Base.getproperty(%5, :flatten)
7 t │ $(Expr(:const, :(StaticArrays.StableFlatten.broadcast_flatten), :(%6)))
8 f │ $(Expr(:latestworld))
9 f └── return %6
10 f 3 ─ using Base: tail
⋮
The return statement is not marked. @aviatesk, any chance that https://github.com/JuliaDebug/LoweredCodeUtils.jl/pull/117 might fix it?
Thanks for fixing my example!
The
returnstatement is not marked
Sure, I will look into it at least by tomorrow.
I didn't know about
@activate. Is there a demo of the workflow somewhere? I've seen the docs, and they're good, but I suspect there's a bit more wisdom around effective usage than they convey.I can back that commit out, if there are other mechanisms to achieve the same aim. Are you saying that
Revise.track(Core.Compiler)should be deprecated?
Sorry for the delay, but I'd like to add some details about this.
Currently, when you use InteractiveUtils.@activate Compiler, the Compiler.jl implementation specified in your environment gets loaded. This also switches the Compiler.jl implementation used by reflection utilities like code_typed to the newly loaded one.
For the actual Compiler.jl implementation, besides a dummy implementation that just redirects to Base.Compiler (https://github.com/JuliaLang/BaseCompiler.jl), we expect the implementation to be used is a local checkout of JuliaLang/julia/Compiler or a specific remote JuliaLang/julia commit.
So when you're hacking on Compiler.jl, using InteractiveUtils.@activate Compiler is generally enough, and I think Revise.track(Core.Compiler) can be deprecated. In most cases, I don't see any benefit in applying changes to the Base.Compiler module and observing the changes for the Compiler development. There might still be situations where it still is potentially useful, like for debugging inference issues that only happen during bootstrap. But this problem itself is quite tricky, and I feel it's better to use simple reflections like Core.eval or Core.println for debugging it, rather than advanced tools like Revise. So in conclusion, I think Revise.track(Core.Compiler) can be deprecated.
@Keno Please let me know if you have anything to add.
For some reason, @KristofferC's example seems to be working fine with the latest combination of Revise/LoweredCodeUtils/Tensors.
julia> using Revise
julia> using Tensors
julia> fieldnames(Tensors.Tensor)
(:data,)
julia> fieldnames(Tensors.Tensor)
(:data, :x)
julia> VERSION
v"1.12.0-rc1.23"
Amusingly/terrifyingly, I have even occasionally triggered the message "Compiling the compiler. This may take several minutes ..." just from using the REPL and making revisions 😱
OK, I have to correct what I said above: it does work, both with the released version of LCU and with https://github.com/JuliaDebug/LoweredCodeUtils.jl/pull/117 (I'm not sure why it didn't before).
But with both I get
Compiling the compiler. This may take several minutes ...
Base.Compiler ──── 11.776275873184204 seconds
and obviously that needs to be fixed.
Well, that was easy. This branch may now be worth trying more seriously (once tests get back to passing).
I thought I had commented, but I guess not
I think
Revise.track(Core.Compiler)can be deprecated
This is very occasionally still useful, when the Compiler in your environment is not the same as the base compiler and you want to test things. So I don't think we need to remove it, we can just teach people about the new, better workflow for Compiler development, and Revise.track can remain an option for power users.
I restored the ability for Revise to call Compiler.bootstrap!() but made it conditional on Revise being sure that it was triggered by changes made by Revise. This is conservative, and automatic revision will stop if methods get invalidated just by new method definitions (e.g., loading packages).
struct revision tests are now passing again (the failure is only in extra tests).
In principle, this could be ready to merge. Please test in real workflows (nightly, as rc1 is currently broken).