test262 icon indicating copy to clipboard operation
test262 copied to clipboard

Only use `phase: parse` for errors that happens while parsing the test entry point

Open nicolo-ribaudo opened this issue 1 year ago • 2 comments

https://github.com/tc39/test262/blob/main/CONTRIBUTING.md#negative

For a file with no dependencies, it's obvious that parse means that the error is thrown when calling ParseModule on its source code.

On the other hand, what happens when there are multiple files involved is less clear. Resolution of module dependencies is not defined in ECMA-262, but in HTML. Before the layering changes from 2022, there was a HostResolveImportedModule host hook that was responsible for loading/resolution of modules. There is also a ResolveExport AO, that might map to this "test262" resolution phase.

When running a test, there four possible phases:

  1. module = ParseModule(): parse the entry module
  2. module.LoadRequestedModules(): resolve, load and parse its dependencies
  3. module.Link(): resolve the various exported and imported bindings
  4. module.Evaluate(): executes the module and its dependencies

Before the 2022 layering changes, 1 and 2 where mixed together from an ECMA-262 point of view.

Looking at current tests, it seems like:

  • (1) is covered by phase: parse
  • (3) is covered by phase: resolution
  • (4) phase: evaluation

There is some confusion about (2):

  • https://github.com/tc39/test262/blob/d2596e265142410a6cb7a01d0ae4bc2fce84ce8b/test/language/import/import-assertions/json-invalid.js throws during (2), because it throws while resolving/loading/parsing ./json-invalid_FIXTURE.json, and it has phase: parse
  • https://github.com/tc39/test262/blob/d2596e265142410a6cb7a01d0ae4bc2fce84ce8b/test/language/module-code/source-phase-import/import-source-binding-name.js throws during (2), because it throws while resolving/loading/parsing <do not resolve>

Using phase: parse for tests thrown during (2) prevents most JavaScript parsers from successfully using test262 as a test suite for parsing tests, since those projects would just call ParseModule( test source ) and would not proceed to (2)/(3)/(4). On the other hand, using phase: resolution for it seems wrong, since module resolution happens outside of ecma262 and thus we are probably testing binding resolution.

I propose that we clarify the following:

  1. module = ParseModule() throws at phase: parse
  2. module.LoadRequestedModules() doesn't have a corresponding phase, and we will make sure to write no tests that throw at this point.
  3. module.Link() throws at phase: resolution
  4. module.Evaluate() throws at phase: evaluation

nicolo-ribaudo avatar Jun 29 '24 07:06 nicolo-ribaudo

I think we should discuss this in the next maintainers meeting.

ptomato avatar Jul 04 '24 16:07 ptomato

Proposal:

  1. Clarify the documentation
  2. Use phase: resolution for test/language/import/import-assertions/json-invalid.js since we couldn't figure a use case to distinguish between your steps 2 and 3
  3. Rewrite test/language/module-code/source-phase-import/import-source-binding-name.js as a fully passing test case by replacing '<do not resolve>' by a real fixture

@nicolo-ribaudo does this make sense to you?

Ms2ger avatar Aug 14 '24 15:08 Ms2ger