Add support for yielding in generalized iteration

[filiptibell]

Lune, which is a standalone Luau script runtime, recently implemented support for web sockets, with the current API looking like this:

local socket = net.socket("ws://localhost:8080")

-- The message will be nil when the socket has closed
repeat
    local message = socket.next()
    if message ~= nil then
        socket.send("Echo: " .. message)
    end
until message == nil

This is a pretty nice API and is mostly ergonomic to use, but it could be made much neater using generalized iteration:

local socket = net.socket("ws://localhost:8080")

-- The loop will end when the socket has closed
for message in socket do
    socket.send("Echo: " .. message)
end

The same pattern could also be applied to a normal http server, similar to the js runtime Deno:

for request in net.serve(8080) do
    request.respondWith("Echo: " .. request.body)
end

These both currently throw errors because the __iter metamethod would need to yield. The compatibility page on the luau-lang site states that this is for performance reasons, but I'm thinking that limiting it to generalized iteration specifically could maybe help with that and also make sense if there is a strong use case.

[jackdotink]

I think this would be a negative change, as it creates inconsistencies. The current rule is that you cannot yield in metamethods. This would change that to you cannot yield in metamethods, except for __iter. This creates a challenge for people learning the language, and is generally inconsistent, especially when the alternative is to just call a function that returns a function that can be iterated over, no need for a metamethod.

[filiptibell]

especially when the alternative is to just call a function that returns a function that can be iterated over, no need for a metamethod.

That doesn't work, though. Modifying the example to do that instead yields indefinitely and never gets any messages:

local socket = net.socket("ws://localhost:8080")

local function iter()
	return function()
		return socket.next()
	end
end

for message in iter() do
	socket.send("Echo - " .. message)
end

[jackdotink]

I would support yielding in iteration, it turns out that this example errors. The more you know.

function iter (i)
	task.wait(1)
	i = i + 1
	if i < 10 then
		return i
	end
end

function loop()
	return iter, 0
end

for i in loop() do
	print(i)
end```

[goldenstein64]

Would it be better to abstract away the iteration part and use a signal/callback-based interface?

-- server.luau
local socket = net.socket("ws://localhost:8080")

socket.received:connect(function(message)
  socket.send("Echo: " .. message)
end)

socket.listen()

-- somewhere in net.socket (but also in rust)
function socket.listen()
  local message = socket.next()
  while message ~= nil do
    socket.received:fire(message)
    message = socket.next()
  end
end

[filiptibell]

Would it be better to abstract away the iteration part and use a signal/callback-based interface?

In many cases, yes, probably. For Lune though I want to provide a simple interface and defaults that are as intuitive as possible to use without wrapping the API in things such as signals. Users can then import the signal library that fits them and make their own wrappers if that's what they want.

Worth noting is that yielding during iteration is something that Roblox seems to want to support, here's an official documentation page suggesting to use it (but that doesn't actually work): https://create.roblox.com/docs/reference/engine/classes/Pages

[m-doescode]

I'm curious as to why we can't just allow yielding in metamethods. Is it some sort of infrastructure problem? Or is it related to performance?

[BenMactavsin]

I'm curious as to why we can't just allow yielding in metamethods. Is it some sort of infrastructure problem? Or is it related to performance?

It's related to performance: https://luau-lang.org/compatibility#lua-52

[zeux]

Note: we should distinguish ability to yield from metamethods, and ability for the iterator function to yield. __iter produces an iterator function that is called once before every iteration of the loop, including the first one; as such, we could maintain that all metamethods don't yield, but can simultaneously allow the iterator function to yield. It's still a little inconsistent in a sense, but much less so than allowing __iter to yield.

That said, there might still be efficiency and implementation concerns with this. We'll look into how viable this is for our implementation.