[New solver] Type aliases that pass generic parameters into the `setmetatable` type function break the solver when used as type annotations

[deviaze]

Tested on:

• luau-analyze --fflags=true with brand new compile of luau-analyze (0.660)
• luau lsp (0.660)

Minimal repro:

--!strict
-- TypeError: Type inference failed to complete, you may see some confusing types and type errors.
type class<S, I> = setmetatable<Struct, {
    __index: I
}>

local Cat: class<{}, {}> = {} -- TypeError: Type '{  }' could not be converted into '{ @metatable { __index: {  } }, {  } }'

Directly using the setmetatable type function doesn't cause this (and produces the expected TypeError):

--!strict
local Cat: setmetatable<{}, {}> = {} -- TypeError: Type '{  }' could not be converted into '{ @metatable {  }, {  } }'

Interestingly, using class as a type cast (instead of an annotation) also works fine:

--!strict
type class<S, I> = setmetatable<S, {
    __index: I,
}>

local c = {} :: class<{}, {}> -- this works

local d = {} :: class<{}, { hi: (self: any, text: string) -> string}>
d:hi("text") -- works; no type error (as intended)

This isn't localized to the __index field, we can repro with __iter as well:

--!strict
-- TypeError: Type inference failed to complete, you may see some confusing types and type errors.
type withiterator<T, Iterator> = setmetatable<T, {
    __iter: Iterator
}>

local x: withiterator<{
    items: {
        [string]: any,
    }
}, (self: any) -> () -> (number, any)> = {} -- TypeError: Type '{  }' could not be converted into '{ @metatable { __iter: (any) -> () -> (number, any) }, { items: { [string]: any } } }'

[aatxe]

Unsure about the immediate cause of constraint solving not completing here, but the type mismatches that you don't expect are another instance of expected types not behaving properly, which @hgoldstein is currently working on.

[hgoldstein]

Closing this out as I believe the main issue (the type solving failed to complete) has been resolved. Feel free to re-open, but I think the type mismatch errors were expected. For example in:

--!strict
type class<S, I> = setmetatable<S, { __index: I }>

local Cat: class<{}, {}> = {}

... _we should error on the last line, as {} does not have a metatable, but we expect a table with a metatable.