Port PUC Lua's test suite to CC

[SquidDev]

@MCJack123 has been working hard at updating Cobalt (CC:T's Lua runtime) to Lua 5.2. However, we're going to have to be very careful that this update doesn't break any existing (and reasonable) programs. For instance, we'll have to preserve getfenv and setfenv in some form.

I think one useful step here would be to copy some of Cobalt's test suite into CC:T. They're currently a mix of home-grown things, and PUC Lua's test suite - we'll need to extract out what's relevant from there and integrate it.

For instance, some useful things include:

• getfenv/setfenv on functions and numbers (stack levels).
• Coroutines (especially yielding within debug hooks and native functions).
• Basic behaviour of the standard library
• Lua 5.2/5.3 features (e.g. __pairs, utf8, new strings)
• Behaviour of os.time (do we do this already??)

However, I think it's fair to ignore things like:

• Cobalt specific regressions
• debug.getinfo's activelines
• Basic language semantics (e.g. Lua's constructs.lua or verybig.lua).

[sir-maniac]

I hope you don't mind my asking... Want any help with this?

[SquidDev]

Yeah, definitely!

Some waffling, because I can't help myself

I'm kinda in a place where I don't really know what approach would be best for this all.

A bit of a prerequisite is to pin down what bits we care about (i.e. what do people use) and what we don't. For instance, ideally we could drop things like table.maxn - I don't think anything relies on it, but need to find out.

That said, most of this can probably be deferred for a while - doesn't impact most of the test suite.

[sir-maniac]

Okay! :grin: Sorry for not getting back to you very soon.

I haven't spent much time looking at all the historic lua code, but am willing to try to look over it. I guess a good start would be to review the forum helps sections and github issues. Any tips for things to look for?

I noticed time formatting might nor be completely compatible with strftime(). If you haven't already I could try setting up a fuzzing test to make sure compatibility is correct and doesn't regress unnoticed in the future. I haven't looked at the existing tests, though.

[SquidDev]

I noticed time formatting might nor be completely compatible with strftime(). If you haven't already I could try setting up a fuzzing test to make sure compatibility is correct and doesn't regress unnoticed in the future.

Oh, that'd definitely be very cool. There's definitely issues with our time code, and probably also some in string.format and the string matching functions, though those may be harder to fuzz.

I haven't spent much time looking at all the historic lua code, but am willing to try to look over it. I guess a good start would be to review the forum helps sections and github issues. Any tips for things to look for?

I think my plan for this is to take a bunch of popular programs for the forums, find which ones still work (those shell.run changes nobbled an awful lot, and 1.13's key codes broke some more). Then run them again with the 5.2 changes (and possibly some other breakages, like removing deprecated 5.1 functions), and see what, if any breaks. Then for breakages, try to write some regression tests.

This is much more grunt work though, so not really expecting other people to be too excited about it :).

[sir-maniac]

Oh, that'd definitely be very cool. There's definitely issues with our time code, and probably also some in string.format and the string matching functions, though those may be harder to fuzz.

I'll take a look the at fuzzing time formatting. Already done some initial research and read the history. It looks like the lua code always uses Locale.ROOT which probably makes things a bit easier.

It looks like the formatting code is complying to the msvcrt version of strftime which is probably what the vast majority of users will compare it to, so I'll try to make fuzz tests comply with that. But maybe ignore some incompatibility issues, since they would already be problems PUC lua, depending on the C library they are compiled against.

[sir-maniac]

I think my plan for this is to take a bunch of popular programs for the forums, find which ones still work (those shell.run changes nobbled an awful lot, and 1.13's key codes broke some more). Then run them again with the 5.2 changes (and possibly some other breakages, like removing deprecated 5.1 functions), and see what, if any breaks. Then for breakages, try to write some regression tests.

This is much more grunt work though, so not really expecting other people to be too excited about it :).

I'll see what I have time for, but it might be fun to try some old source code out, get a feel for how things are used.

[sir-maniac]

Compatibility issues discussed in #790:

SquidDev commented:

For reference:

$ lua5.1
Lua 5.1.5  Copyright (C) 1994-2012 Lua.org, PUC-Rio
> =("%d"):format(0/0)
-9223372036854775808

$ lua5.3
Lua 5.3.6  Copyright (C) 1994-2020 Lua.org, PUC-Rio
> =("%d"):format(0/0)
stdin:1: bad argument #1 to 'format' (number has no integer representation)

Meanwhile Cobalt returns 0.

---

sir-maniac commented:

Yeah, the main reason for that is the difference in casting.

(long)doubleValue in c just converts the bits to a long, in java, it tries to do the integer equivilent, which isn't possible, so it silently returns 0.

The java equivilent cast is Double.doubleToLongBits(doubleVal).

[SquidDev]

I'm no longer planning to do this within CC:T. Instead, we're maintaining our own similar test suite within Cobalt - this has the added advantage that we can run it against normal Lua implementations to check conformance.