"Tuple field type cannot be Union{}" error in all 1.10 versions

[aplavin]

I've been running some package tests on the upcoming julia version (1.10rc) and noticed this error newly introduced there, compared to 1.9:

julia> Tuple{Union{}}
# 1.9:
Tuple{Union{}}
# 1.10:
ERROR: Tuple field type cannot be Union{}

Not sure if introduced deliberately or as a side-effect of some other change, but it breaks perfectly working code. And I don't see it anywhere in Tuple docs that it shouldn't support all element types.

The above is an MWE, and below is the simplified situation where I actually encountered it:

julia> X = StructArray(a=Union{}[], b=[]);
# 1.10 throws error right here
# 1.9 lets you proceed just fine

# can work with the non-Union{} column:
julia> X.b
Any[]

# can retrieve the Union{} column as well, of course cannot access its elements
julia> X.a |> typeof
Vector{Union{}} (alias for Array{Union{}, 1})

[N5N3]

This is a deliberate change, see #49111.

[vtjnash]

You didn't include a stacktrace, but it looks like this calls collect_structarray, which calls Core.Compiler.return_type, which is a private API in a private package with no documentation or stability guarantees. The closest actual API is Base.promote_op() which should work for this and is documented as such (though using it does not quite accurately meet the AbstractArray or collect APIs--c.f. the implementation of those in Base and how it tries to avoid them).

https://github.com/JuliaArrays/StructArrays.jl/blob/99f05561cab19fb23f478a12a8429764871ccff3/src/collect.jl#L44

help?> Core.Compiler.return_type
  │ Warning
  │
  │  The following bindings may be internal; they may change or be removed in future versions:
  │
  │    •  Core.Compiler
  │
  │    •  Core.Compiler.return_type

  No documentation found for private symbol.

[aplavin]

The MWE doesn't depend on return_type at all. And I don't think the structarray example does either.

See:

julia> arrs = (Union{}[], Int[])

# works on 1.9 – broken on 1.10rc
julia> Tuple{map(eltype, arrs)...}
ERROR: Tuple field type cannot be Union{}

[vtjnash]

Not all apply_type expressions are meaningful. They can return errors

[aplavin]

Creating a Tuple type with arbitrary types inside seems to make total sense, that's one of the most fundamental Julia types. I haven't seen anywhere in the docs that Tuple only supports a subset of Julia types. And this limitation does sound strange, doesn't it?

[aplavin]

So, should this error be removed? Both:

• this change looks unambiguously breaking, perfectly working code that doesn't rely on internals stops working
• the behavior itself is totally sensible, even aside from the first point; surely one can create a Tuple type with any element types, no exceptions are documented and Union{} as element type does work elsewhere.

[rdboyes]

Running into this error using density() from AlgebraOfGraphics as well - a simple MWE from here: https://github.com/MakieOrg/AlgebraOfGraphics.jl/issues/472#issuecomment-1859060029

julia> draw(
           data((x=randn(100), y=randn(100))) *
           mapping(:x, :y) *
           AlgebraOfGraphics.density() *
           visual(Contour)
       )
ERROR: Tuple field type cannot be Union{}
Stacktrace:
  [1] map(f::Function, d::Dictionaries.Indices{Union{}})
    @ Dictionaries ~/.julia/packages/Dictionaries/7aBxp/src/map.jl:91
  [2] unnest(vs::Vector{@NamedTuple{}}, indices::Dictionaries.Indices{Union{}})
    @ AlgebraOfGraphics ~/.julia/packages/AlgebraOfGraphics/tbMEb/src/algebra/layer.jl:81
  [3] unnest_dictionaries(vs::Vector{@NamedTuple{}})
    @ AlgebraOfGraphics ~/.julia/packages/AlgebraOfGraphics/tbMEb/src/algebra/layer.jl:84
  [4] map(f::AlgebraOfGraphics.var"#193#194"{@NamedTuple{…}}, processedlayer::ProcessedLayer)
    @ AlgebraOfGraphics ~/.julia/packages/AlgebraOfGraphics/tbMEb/src/algebra/layer.jl:101
  [5] (::AlgebraOfGraphics.DensityAnalysis{…})(input::ProcessedLayer)
    @ AlgebraOfGraphics ~/.julia/packages/AlgebraOfGraphics/tbMEb/src/transformations/density.jl:30
  [6] call_composed
    @ Base ./operators.jl:1045 [inlined]
  [7] call_composed
    @ Base ./operators.jl:1044 [inlined]
  [8] (::ComposedFunction{AlgebraOfGraphics.Visual, AlgebraOfGraphics.DensityAnalysis{…}})(x::ProcessedLayer)
    @ Base ./operators.jl:1041
  [9] process(layer::Layer)
    @ AlgebraOfGraphics ~/.julia/packages/AlgebraOfGraphics/tbMEb/src/algebra/processing.jl:102
 [10] iterate(g::Base.Generator, s::Vararg{Any})
    @ Base ./generator.jl:47 [inlined]
 [11] collect(itr::Base.Generator{Layers, typeof(AlgebraOfGraphics.process)})
    @ Base ./array.jl:834
 [12] map
    @ ./abstractarray.jl:3310 [inlined]
 [13] ProcessedLayers(a::Layer)
    @ AlgebraOfGraphics ~/.julia/packages/AlgebraOfGraphics/tbMEb/src/algebra/layers.jl:41
 [14] compute_axes_grid(d::Layer; axis::@NamedTuple{}, palettes::@NamedTuple{})
    @ AlgebraOfGraphics ~/.julia/packages/AlgebraOfGraphics/tbMEb/src/algebra/layers.jl:114
 [15] compute_axes_grid
    @ ~/.julia/packages/AlgebraOfGraphics/tbMEb/src/algebra/layers.jl:110 [inlined]
 [16] compute_axes_grid(fig::Figure, d::Layer; axis::@NamedTuple{}, palettes::@NamedTuple{})
    @ AlgebraOfGraphics ~/.julia/packages/AlgebraOfGraphics/tbMEb/src/algebra/layers.jl:100
 [17] compute_axes_grid
    @ ~/.julia/packages/AlgebraOfGraphics/tbMEb/src/algebra/layers.jl:97 [inlined]
 [18] #241
    @ ~/.julia/packages/AlgebraOfGraphics/tbMEb/src/draw.jl:21 [inlined]
 [19] update
    @ ~/.julia/packages/AlgebraOfGraphics/tbMEb/src/draw.jl:10 [inlined]
 [20] plot!(fig::Figure, d::Layer; axis::@NamedTuple{}, palettes::@NamedTuple{})
    @ AlgebraOfGraphics ~/.julia/packages/AlgebraOfGraphics/tbMEb/src/draw.jl:21
 [21] plot!
    @ ~/.julia/packages/AlgebraOfGraphics/tbMEb/src/draw.jl:16 [inlined]
 [22] (::AlgebraOfGraphics.var"#245#246"{…})(f::Figure)
    @ AlgebraOfGraphics ~/.julia/packages/AlgebraOfGraphics/tbMEb/src/draw.jl:48
 [23] update
    @ AlgebraOfGraphics ~/.julia/packages/AlgebraOfGraphics/tbMEb/src/draw.jl:10 [inlined]
 [24] #draw#244
    @ AlgebraOfGraphics ~/.julia/packages/AlgebraOfGraphics/tbMEb/src/draw.jl:47 [inlined]
 [25] draw(d::Layer)
    @ AlgebraOfGraphics ~/.julia/packages/AlgebraOfGraphics/tbMEb/src/draw.jl:44
 [26] top-level scope
    @ REPL[7]:1
Some type information was truncated. Use `show(err)` to see complete types.

[vtjnash]

As with the previous person, this is due to the Dictionaries package using the disallowed private symbol Core.Compiler.return_type. Open an issue there and link https://github.com/JuliaLang/julia/issues/52385#issuecomment-1839431579?

[aplavin]

This issue (the one discussed here, don't know about AoG) is independent on return_type() though, but due to a breaking change in Julia 1.10. Even more, it's breaking completely sensible behavior of being able to represent Tuple{T} for T = Union{}.

There's nothing fundamentally bad with breaking changes, they should just be clearly communicated. For now, Julia promises no breaking changes in 1.x, which this clearly contradicts.

[nsajko]

Dup of #51950 EDIT: although the example there was perhaps more benign.

[nsajko]

The empty union, Union{}, is the type with no instances, thus Tuple{Union{}}, and any other tuple type with an empty union field, is also the type with no instances, so I guess we should have Tuple{Union{}} == Union{} hold. I don't see how it makes sense for Tuple{Union{}} to error.

I admit this might be difficult to fix, though.

[vtjnash]

Having no instances is not the same as having no subtypes. Those are 2 orthogonal properties.

We could try to return Union{} here, as that was the first attempt before making it an error, but much code behaved poorly with that, so it would be a breaking change. Making it an error--for this case which was already buggy--was not a breaking change, since packages had always been told not to call the Core.Compiler functions.

[nsajko]

Both Tuple and Union are part of the public API. So it seems like Tuple{Union{}} is public API, regardless of any Core.Compiler internals, no?

[aplavin]

was not a breaking change, since packages had always been told not to call the Core.Compiler functions.

Not sure why Core.Compiler.return_type is being brought up here over and over, it's completely unrelated to the issue.

[aplavin]

https://github.com/JuliaLang/julia/issues/51950 can be a "duplicate" from the internals PoV (function signatures are implemented as tuples), but for a user these are two different scenarios. f(::Union{}) = ... mentioned in https://github.com/JuliaLang/julia/issues/51950 is a method that cannot be called at all (right?), while Tuple{Union{}} can sometimes arise in generic code without any bugs or internals usage.

[palday]

Making it an error--for this case which was already buggy--was not a breaking change, since packages had always been told not to call the Core.Compiler functions.

@vtjnash I've opened a PR against Dictionaries.jl that only uses public API and this problem still arises, so this is definitely not an issue arising relying on compiler internals.

[andyferris]

Union{} is always an interesting case, and given that Tuple parameters are covariant I can see that the behavior of Tuple{Union{}} might be different than Vector{Union{}} or wherever the parameter is invariant. E.g. it might be OK to instantiate an empty Vector{Union{}} but not a Tuple{Union{}} (which would be similar to trying to instantiate a Union{}, which is plainly impossible).

Still - in terms of the implementation, having Tuple{Union{}} become Union{} seems better for users than a runtime error. Especially for generic code.

As for Core.Compiler.return_type, I'm honestly happy to use whatever works. Note that sometimes whatever logic is used in Base in methods such as map(f, ::Vector) needs to be replicated in packages for other data structures. In Julia 1.10 this ultimately uses Base.@default_eltype to infer the eltype (which itself uses Core.Compiler.return_type).

[LilithHafner]

This seems like legitimate Julia code that does not depend on internals, though I don't think I've ever had a use-case for Union{}[] so I don't know how reasonable this usage is.

julia> eagerzip() = error()
eagerzip (generic function with 1 method)

julia> function eagerzip(args::AbstractArray...)
           allequal(axes.(args)) || throw(DimensionMismatch())
           Base.require_one_based_indexing(args...)
           res = similar(first(args), Tuple{eltype.(args)...})
           for i in eachindex(args...)
               res[i] = getindex.(args, i)
           end
           res
       end
eagerzip (generic function with 2 methods)

julia> eagerzip([1,2,3], [5,6,7])
3-element Vector{Tuple{Int64, Int64}}:
 (1, 5)
 (2, 6)
 (3, 7)

julia> eagerzip(Int[], [])
Tuple{Int64, Any}[]

julia> eagerzip(Union{}[], [])
Tuple{Union{}, Any}[] # 1.9
ERROR: Tuple field type cannot be Union{} # 1.10
Stacktrace:
 [1] eagerzip(::Vector{Union{}}, ::Vararg{AbstractArray})
   @ Main ./REPL[201]:4
 [2] top-level scope
   @ REPL[204]:1

[KristofferC]

Looks effectively the same as the example in https://github.com/JuliaLang/julia/issues/52385#issuecomment-1839518854 (except much longer)?

[LilithHafner]

Yep! It's the same, just with a bit more motivation. The first example in the OP was, by itself, a valid regression report IMO.

[vtjnash]

Do you have an example of that? Arguably Union{}[] would have been the correct return there (which is what we fixed out to return in other code where it came up)

[LilithHafner]

No, I don't have an example of this breaking anything "in the wild" that does not depend on internals. If nobody else has such an example either, then we can call it a minor change and be done with it, but it is technically breaking.

[jariji]

Irrespective of semantic versioning, I don't see how it follows from any of the documented type rules that this should fail, so the error would be a special case, which isn't great from a user perspective.

[vtjnash]

it follows from any of the documented type rules that this should fail

It follows from the rule that the intersection of Tuple{T} and Tuple{S} is empty if the intersection of T and S is empty. That is a rule that subtyping has always had, so that is why this wasn't a major breaking change, as it only sets out to align the rest of the system with the existing rule. As for being a special case, this implementation is equivalent to adding a lower bound on the typevar for Tuple, which is not a particularly special case. Although it does also happen to implement the oft-requested feature of being able to prohibit a type var from being exactly Union{} although without yet making that feature very generally accessible.

but it is technically breaking

OT, but every change, including bugfixes, are technically breaking. But semantic versioning is mostly about not changing the result beyond the limits of what was promised. I actually wonder if there is an argument to be made that most changes from semi-working -> error or error -> working is allowable by semantic versioning therefore, since in neither case does a working program get a different return value.

[aplavin]

No, I don't have an example of this breaking anything "in the wild" that does not depend on internals. If nobody else has such an example either, then we can call it a minor change and be done with it

Aren't there enough examples in this thread already? I'm not sure why "internals" are even brought up here, as Core.Compiler is completely unrelated to this issue.

Julia docs say that

As per SemVer, code written for v1.0 will continue to work for all future LTS and Stable versions.

I've always interpreted that (and want to continue doing so...) that any code (not relying on internals/experimental) will continue to work. Not "only code that julia devs explicitly approve".

Moreover, as @jariji also points out, this is not just a breaking change in the abstract – it breaks totally sensible behavior, not a weird historical quirk.

Vector or AbstractVector with eltype == Union{} is a perfectly fine thing in Julia. It's also a natural thing to put such a vector into a StructArray. And this doesn't work anymore in 1.10.

Nothing in the Tuple/NamedTuple documentation suggests that they only support a subset of Julia types. Like, I can create another type with any parameter that I want:

julia> struct S{T}
       t::T
       end

julia> S{Union{}}
S{Union{}}

but not Tuple or NamedTuple for some reason.

[LilithHafner]

Triage thinks that this is a breaking change.

It might be acceptable to keep it, but we would need a strong example of the havoc that allowing Tuple{Union{}} would cause. The breakage here seems more bad than the original issue https://github.com/JuliaLang/julia/issues/32392.

Triage didn't see why https://github.com/JuliaLang/julia/issues/32392 is such a big deal, perhaps it would be worth expanding on what led to that report.

Possible course of action: release 1.10.1 with Tuple{Union{}} constructible

[JeffBezanson]

I know there are other problems with the type system that removing Tuple{Union{}} helps avoid, but it seems like a very "early" point to throw an error. Programs might very well try things like Tuple{eltype(a)} and it's not clear how to change that code to avoid the error. Can we push the problem farther downstream, closer to an "actual problem" and not just the existence of the type object?

[nsajko]

Issues around Tuple{Union{}} were actually discussed all the way back in 2018, in the Julia subtyping paper (Julia Subtyping: A Rational Reconstruction. Proceedings of the ACM on Programming Languages, 2018, 27, ⟨10.1145/3276483⟩. ⟨hal-01882137⟩).

We propose an alternative design. The type Tuple{Union{}} (or, more generally, any tuple type containing Union{} as one of its components) is not inhabited by any value, and dispatch-wise it behaves as Union{}. However, neither Julia 0.6.2 nor our formalization can prove it equivalent to Union{} because the judgment Tuple{Union{}}<:Union{} is not derivable: following Julia 0.6.2 semantics, the lift_union function does not lift empty unions out of tuples. Extending lift_union to lift empty unions, thus rewriting types such as Tuple{Union{}} into Union{}, is straightforward; the resulting subtype relation is not affected by the transitivity problem described above. We have modified our reference implementation along these lines. Testing over the real-world workloads does not highlight differences with the standard subtype relation, suggesting that this change does not impact the programming practice. However, this alternative design has implementation drawbacks. Assuming that a Tuple{t} type in a program where t is unknown yields a 1-element tuple type becomes incorrect, making dataflow analyses more complex. Also, uniqueness of the bottom type is lost, and testing if a type is bottom (a test that occurs surprisingly often in Julia code-base) becomes slower. These tradeoffs are being investigated by Julia developers.

and in another place in the paper:

Unprovable judgments. Julia’s subtype algorithm, and in turn our formalization, cannot prove all judgments expected to hold. For instance it cannot prove: (Tuple{T} where String <:T <:Int) <: Union{ } or Tuple{Union{ }} <: Union{ } despite all these types having no elements (the type on the left-hand side being a valid Julia type).

The latter quote also ties into #24179, which was closed for some reason.

[tpapp]

@JeffBezanson: FWIW, I would always prefer

Tuple{eltype(a)}

to error if eltype(a) === Union{}, because that is a seriously broken eltype and I would prefer to know as early as possible.

To take this further, I would also like all types to error if a type parameter is Union{}, eg Vector{Union{}} etc.

I get why having Union{} at the bottom is nice, for the compiler and just to have a type that is a subtype of everything, but it leaking all over the type system is not worth it IMO.

Incidentally, we should also make

SomeParametricType{Union{}} <: (SomeParametricType{T} where T<:SomeConcreteType)

false to be consistent, if the left hand side errors.

[nsajko]

if eltype(a) === Union{}

That can never hold, Union{} can't have instances, i.e. a isa Union{} holds for no a. EDIT: sorry, I thought you wrote typeof when you wrote eltype.

[tpapp]

I agree that it should not hold, but cf

julia> v = Vector{Union{}}(undef, 1)
1-element Vector{Union{}}:
 #undef

julia> eltype(v)
Union{}

I think that there are two consistent solutions:

1. never allow Union{} as a type parameter, so Tuple{Union{}} and friends error, and thus don't include it in Foo{T} where T etc,

2. allow Union{} as a type parameter, and then allow it everywhere, incl Tuple, and then also Foo{Union{}} is a subtype of the above where clause.