New type solver refuses to accept string literal despite being one of the requested types

[melindatrace]

type Form = "do-not-register" | (() -> ())

local function observer(register: () -> Form) end

observer(function()
	if math.random() > 0.5 then
		return "do-not-register" -- TypeError: Type 'string' could not be converted into '"do-not-register" | (() -> ())' Luau(1000)
	end
	return function() end
end)

Workaround:

type Form = "do-not-register" | (() -> ())

local function observer(register: () -> Form) end

observer(function(): Form
	if math.random() > 0.5 then
		return "do-not-register" -- OK
	end
	return function() end
end)

This does not happen to code like this:

local function min(): "str" | (() -> ())
    return "str"
end

[biseson]

TL;DR The newer type system expects the more precise literal type, but without an explicit type annotation, it defaults to string.

It's like that because, in Luau's new type solver, type inference for anonymous functions may not always correctly infer a specific string literal type when it appears in a union like "do-not-register" | (() -> ()), the type system infers "do-not-register" as a broader string type; not matching the expected literal type.

This likely occurs because the return type of the anonymous functions is initially inferred as a string, even though "do-not-register" is a string literal, that matches the expected "do-not-register".

[hgoldstein]

Hello! Thank you for the report. Just wanted to check in to say I'm working on this class of issues. In this case we'd like the function passed into observer to have the correct type inferred, as it's clear the intent is for the given function to have the type () -> Form. There's two issues here:

• We're not properly "pushing" the type of () -> Form into the lambda: the other comment is correct that we end up inferring a broader string type (as we try to do so for literals unless we discover we need it to be a singleton type)
• There's an outstanding bug where instead of inferring return types based on the union of return statements, we instead do so based on picking the first return statement and assuming that's the final return type.