segfault when pre-compiling a modules that use NamedTuples

[mdavezac]

When creating a simple package and trying to pre-compile:

__precompile__()
module MyPackage
using NamedTuples

const a_named_tuple = @NT(a = 1, b = 2)
end

I get the following error:

julia> using MyPackage
INFO: Recompiling stale cache file XXX/julia/lib/v0.5/MyPackage.ji for module MyPackage.
WARNING: eval from module NamedTuples to MyPackage:
Expr(:type, false, Expr(:<:, Expr(:curly, :_NT_a_b, :T1, :T2)::Any, :NamedTuple)::Any, Expr(:block, Expr(:::, :a, :T1)::Any, Expr(
:::, :b, :T2)::Any)::Any)::Any
  ** incremental compilation may be broken for this module **


signal (11): Segmentation fault: 11
while loading no file, in expression starting on line 0
jl_deserialize_value_ at /Users/osx/buildbot/slave/package_osx10_9-x64/build/src/dump.c:1743
jl_deserialize_datatype at /Users/osx/buildbot/slave/package_osx10_9-x64/build/src/dump.c:1238 [inlined]
jl_deserialize_value_ at /Users/osx/buildbot/slave/package_osx10_9-x64/build/src/dump.c:1736
jl_deserialize_value_ at /Users/osx/buildbot/slave/package_osx10_9-x64/build/src/dump.c:1733
jl_deserialize_value_ at /Users/osx/buildbot/slave/package_osx10_9-x64/build/src/dump.c:1668
jl_deserialize_value_ at /Users/osx/buildbot/slave/package_osx10_9-x64/build/src/dump.c:1485
_jl_restore_incremental at /Users/osx/buildbot/slave/package_osx10_9-x64/build/src/dump.c:2532
jl_restore_incremental at /Users/osx/buildbot/slave/package_osx10_9-x64/build/src/dump.c:2580
_include_from_serialized at /Applications/Julia-0.5.app/Contents/Resources/julia/lib/julia/sys.dylib (unknown line)
_require_from_serialized at /Applications/Julia-0.5.app/Contents/Resources/julia/lib/julia/sys.dylib (unknown line)
require at /Applications/Julia-0.5.app/Contents/Resources/julia/lib/julia/sys.dylib (unknown line)
jlcall_require_22363 at /Applications/Julia-0.5.app/Contents/Resources/julia/lib/julia/sys.dylib (unknown line)
jl_call_method_internal at /Users/osx/buildbot/slave/package_osx10_9-x64/build/src/./julia_internal.h:189 [inlined]
jl_apply_generic at /Users/osx/buildbot/slave/package_osx10_9-x64/build/src/gf.c:1942
jl_apply at /Users/osx/buildbot/slave/package_osx10_9-x64/build/src/./julia.h:1392 [inlined]
eval_import_path_ at /Users/osx/buildbot/slave/package_osx10_9-x64/build/src/toplevel.c:402
eval_import_path at /Users/osx/buildbot/slave/package_osx10_9-x64/build/src/toplevel.c:429 [inlined]
jl_toplevel_eval_flex at /Users/osx/buildbot/slave/package_osx10_9-x64/build/src/toplevel.c:480
jl_toplevel_eval_in_warn at /Users/osx/buildbot/slave/package_osx10_9-x64/build/src/builtins.c:590
eval at /Applications/Julia-0.5.app/Contents/Resources/julia/lib/julia/sys.dylib (unknown line)
jlcall_eval_19762 at /Applications/Julia-0.5.app/Contents/Resources/julia/lib/julia/sys.dylib (unknown line)
jl_call_method_internal at /Users/osx/buildbot/slave/package_osx10_9-x64/build/src/./julia_internal.h:189 [inlined]
jl_apply_generic at /Users/osx/buildbot/slave/package_osx10_9-x64/build/src/gf.c:1942
eval_user_input at ./REPL.jl:64
unknown function (ip: 0x322bfc0e6)
jl_call_method_internal at /Users/osx/buildbot/slave/package_osx10_9-x64/build/src/./julia_internal.h:189 [inlined]
jl_apply_generic at /Users/osx/buildbot/slave/package_osx10_9-x64/build/src/gf.c:1942
macro expansion at ./REPL.jl:95 [inlined]
#3 at ./event.jl:68
unknown function (ip: 0x322bf2fdf)
jl_call_method_internal at /Users/osx/buildbot/slave/package_osx10_9-x64/build/src/./julia_internal.h:189 [inlined]
jl_apply_generic at /Users/osx/buildbot/slave/package_osx10_9-x64/build/src/gf.c:1942
jl_apply at /Users/osx/buildbot/slave/package_osx10_9-x64/build/src/./julia.h:1392 [inlined]
start_task at /Users/osx/buildbot/slave/package_osx10_9-x64/build/src/task.c:253
Allocations: 1878057 (Pool: 1877182; Big: 875); GC: 1
zsh: segmentation fault  julia

Any call to @NT seem to have the same effect. I'm using julia v0.5 on OS/X, installed from homebrew.

[davidanthoff]

@mdavezac Is this still happening? If yes, then this is some bug in julia itself, it should really never segfault and maybe @JeffBezanson could look at it?

[mdavezac]

Yes, let me check again. This was in in 0.5.0, and I've just upgraded to 0.5.1. I had other issues with precompilations in 0.5.0.

[mdavezac]

Unfortunately, this is also true in 5.1.

[mdavezac]

As well as 0.5.2 and 0.6-rc1

[JeffBezanson]

The incremental compilation may be broken for this module message seems to be accurate here. It's probably impossible to precompile a module that contains a NamedTuple value, since we don't know where its type came from. A possible workaround is to set the constant from __init__ instead of at the top level.

We could potentially fix this in this package by creating the NamedTuple types in the modules using them, rather than in the NamedTuples module (which I think it used to do at one point). However that would have the cost of these types no longer being unique; there would be multiple "copies" of the same named tuple types in various modules.

[davidanthoff]

But should it segfault? I'm just wondering whether there could be a more graceful error message :)

I don't think creating the types in different modules is a good idea, the downside you describe seems too real.

I was just going to say that hopefully we can change the whole implementation of this once julialang/julia#1974 is done, but I just saw (with almost some tears in my eyes) that you moved that to 2.0+... Totally understand the need for a schedule, but man, I'd love to see that implemented...

In particular, I think it would allow us to get rid of the whole type generation, right? We could just have

struct NamedTuple{T,N}
    values::T
end

Where N would be a tuple of Val{Symbol} of the names. For example, a named tuple with fields name, children and age would be of type NamedTuple{Tuple{String,Int64,Float64},Tuple{Val{:name},Val{:children},Val{:age}}}.

I believe that kind of thing would both solve the precompile issue, and also allow us to implement a type stable merge (#4), which is the main, main roadblock right now for Query.jl to get really competitive with something like dplyr.

[JeffBezanson]

The segfault I can fix. Agree with the rest of your assessment.

[vtjnash]

I thought the big warning message was a pretty good error message, considering how hard it can be to detect this sort of corruption (and how easy it is to bypass that message). NamedTuples should probably declare __init__() = __precompile__(false) so that Base can give the right error message with confidence rather than relying on the current heuristic detection.

[mdcfrancis]

This module is urgently used in a number of ore-compiled modes which will now fail with this change. We may have to look at adding back the local module compilation.

[juliohm]

I just had the same issue, was considering NamedTuples.jl as dependency but I will wait until Julia v0.7.