Add script to generate standard property tests

[anba]

0166442283bbcd242d4cd23ee3a51d3a42a84484

• Add verifyBuiltinFunction and verifyBuiltinConstructor. Both functions are used in the generated tests.
• The new harness functions use Reflect and Proxy to identify constructors without invoking any extra MOP methods. The existing isConstructor harness function performs an extra [[Get]] access from [[Construct]] → OrdinaryCreateFromConstructor → GetPrototypeFromConstructor → Get. The new approach avoids this extra [[Get]].

9ad1b2b6a04bf73db50d061c38b7342331e14571

• Change the existing generator to not delete generated tests from other generators.
• This change is necessary because otherwise CI complains when regenerating tests from "make.py".

c5351e479498d8b88832d2a4b69b290d103e6067

• Python scripts to generate the standard property tests.
• The tests are generated from the actual spec files. The spec "parsing" is mostly regular expression matching against known spec phrases and some additional heuristics to properly identify partial spec files from proposals.
• The property test generator was tested against ECMA-262, ECMA-402, and all stage 3 and stage 2.7 proposals.

cfe8c2ad15ac00ee4bbffa135eb12050c1f692ee

• Example commit showing generated property tests for the Number built-in.

b012d8ed3e99cd70733348682c087c6ad57270b9

• Example commit showing generated property tests for ECMA-402 overrides of the Number built-in.

933d746746b14191deb717749b1ac16c49062b40 3b224c4aa8fd5c40e2e668fad9df88b9b81f3e8e

• Remove obsolete tests which are now covered by the generated tests.

8b5cdbc6a00e143d59715de15db599060df486dd

• Example commit showing generated property tests for a spec proposal.

[anba]

I had held off on helpers like this because of dependency upon Reflect and especially Proxy, [...]

I don't actually know if there are any test262 consumers which pull the latest test262 commits, but don't support Reflect or Proxy. But I guess we'll find out if there are any complaints. :smile:

[...] although I do have concerns about the maintenance burden of the assumptions and parsing aspects of property-test-generator, which seem underdocumented. Maybe some of it should be pulled out of test262 for broader use and distinct maintenance?

Hmm, I understand your proposal, but I'm not sure it's useful right now to invest time making the parser more reusable for others, if there aren't yet any other projects which could benefit from it. Should this ever happen, I'm open to move the parser into a separate project, so it's easier to coordinate and plan how to extend it to match the project needs of others.

I don't think we need to spend much time on maintenance, because the specification seems to be in a fairly stable shape w.r.t. how properties and functions are defined. When trying the parser on past ECMA-262 editions, I didn't observe any issues which can't be easily fixed through small code updates:

Edition	Parse Success?	Notes
2025	:x:	Fails for Iterator.prototype.constructor: https://github.com/tc39/ecma262/pull/3558
2024	:ok:	Prints a warning for unsupported old @@symbol notation.
2021-2023	:ok:	Prints a warning for unsupported value definitions.
2016-2020	:x:	Fails for old host-defined global prototype definition: https://github.com/tc39/ecma262/pull/1951

[ptomato]

I didn't review it as closely as Richard. The first few commits seem fine to go ahead with, maybe in a separate PR so they're not blocked while resolving the rest of the comments.

I have to say that I'm somewhat skeptical of running regular expressions on the ECMA-262 prose and generating tests from that directly. Personally, I'd prefer a config file with all the entries for which to generate property tests. The config file could still be generated/updated with a fancy script that reads ECMA-262 if you like, but at least that way there'd be a human in the loop and a clear overview of what changed whenever we update it.