The usage of python.args is inconsistent with Lua varargs semantics

[guidanoli]

Introduction: With the recent introduction of python.args, a subtle semantic defect relating to Lua varargs has leaked. Moreover, the performance of Python calls in Lua (with and without python.args) would be greatly improved as a side effect of a more restricted usage of python.args.

Reasoning: Currently, in Lua, the ellipsis token (...) is used for representing varargs inside functions.

-- talk: example of varargs in functions
function talk(name, ...)
   print(name, 'says', ...)
end

talk("Bob", "the answer is", 6 * 7)
-- prints "Bob says the answer is 42"

talk("Bob")
-- prints "Bob says"

Additionally, the ... expression can be used in table constructors.

-- pack: example of varargs in table constructors
function pack(...) return {...} end

t = pack(1, 2, 3)
for _, v in ipairs(t) do
   print(v)
end
-- prints 1 2 3

However, with multiple varargs, only the last one is expanded.

-- pack3: example of multiple varargs
function pack3(...) return {..., ..., ...} end

t = pack3(1, 2, 3)
for _, v in ipairs(t) do
   print(v)
end
-- prints 1 1 1 2 3

Analogously, function calls also have their return values expanded.

-- get123: example of multiple return values
function get123() return 1, 2, 3

print(get123())
-- prints 1 2 3

However, with multiple function calls, only the last one is expanded.

print(get123(), get123(), get123())
-- prints 1 1 1 2 3

In contrast, the current handling of python.args allows multiple instances to expand in the same call.

get123 = python.args{1, 2, 3}

python.builtins.print(get123, get123, get123)
-- prints 1 2 3 1 2 3 1 2 3

Notice the dissonance between this example and the previous, in pure Lua.

Proposed solutions:

One option would be to expand python.args only if it is the last argument of a function call. This solution is most cohesive with the current Lua varargs semantics, since python.args would only be expanded if last. As a side bonus, detecting and handling python.args would have significant performance gains.

A second option would be to restrict the usage of python.args for keyword arguments only. This would not clash with current Lua semantics, since it does not support keyword arguments. Note that table unpacking is possible anyways with the table.unpack function in standard Lua, so we would not be loosing any functionality for unpacking positional arguments. Furthermore, this would also bring significant performance benefits to the current implementation of Python calls in Lua.

[scoder]

Hmm. I don't really see this as a big issue. The python.args function is a very explicit way to say "I want Python semantics for this call, with args and kwargs". Remember that we introduced this function because of the current behaviour for table arguments.

To me, the Lua semantics look like a gotcha here (it loses data, after all), so I guess that once Lua users ran into them and learned about them, they'd probably expect them. But I don't see a requirement to follow Lua here. And I'm also not completely sure that we are "following Lua" here. I don't think the examples above are completely comparable.

[guidanoli]

On the contrary, Lua users expect packed arguments to only be unpacked only if they're last, and bringing the Python behavior to Lua calls would be a source of unnecessary confusion, ergo a "gotcha".

This is natural to Lua users:

python.builtins.print(2, 3, python.args{4, 5, 6, sep=', '})
-- prints 2, 3, 4, 5

While this is not natural to Lua users:

python.builtins.print(python.args{4, 5}, python.args{6, sep=', '})
-- prints 2, 3, 4, 5

In the use case you gave on #177, this would be even more dramatic:

python.builtins.print(A, B, C, D)
-- currently, we need to know if A, B, C, D are python.args in order to estimate the number of positional arguments
-- in pure Lua, we only need to know if D is python.args to estimate the number of positional arguments (3 + #D)

I reiterate here the importance of keeping Lua code natural to the Lua programmer.

[guidanoli]

We can, of course, rename python.args to python.unpack or something related to give the user a cue for the same Lua semantics as table.unpack but for Python arguments.