What is the purpose of test/staging/?

[linusg]

#3591 quietly introduced the test/staging/ directory without explaining its purpose. Since then more tests have been added there, but I'm failing to find any documentation about it.

• Are test262 runners expected to include or exclude tests in this directory by default?
• Are tests in this directory expected to pass in an engine with a correct implementation of the latest (draft) spec these tests are targeting?
• What are the requirements for someone writing tests to add them to staging, and when can they be moved out of staging?

We're seeing an unusal high failure rate (75% not passing overall) for that directory, so some clarification would be greatly appreciated!

[ljharb]

The intention is for it to be for implementation-contributed tests that don’t yet meet the bar for being full test262 tests. That way, implementations are less likely to duplicate work.

My hope is that implementations don’t run those tests (except for the ones they’ve contributed), but instead help migrate the tests to the proper place, and run them from there.

cc @syg for a better explanation (which we should probably add to the readme)

[syg]

The intention is for it to be for implementation-contributed tests that don’t yet meet the bar for being full test262 tests. That way, implementations are less likely to duplicate work.

The intention is to get tests that are correct and runnable running across implementations ASAP, without having to massage them to conform to test262 directory structures, frontmatter style, and so forth. Eventually they should be massaged and merged into the main test262 suite.

My hope is that implementations don’t run those tests (except for the ones they’ve contributed), but instead help migrate the tests to the proper place, and run them from there.

No, the intention is explicitly for all implementations to run tests in tests/staging/ to ensure interop as early as possible.

To @linusg's specific questions:

• Are test262 runners expected to include or exclude tests in this directory by default?

Expected to include.

• Are tests in this directory expected to pass in an engine with a correct implementation of the latest (draft) spec these tests are targeting?

Yes.

• What are the requirements for someone writing tests to add them to staging, and when can they be moved out of staging?

Requirement for adding a test to staging is that it

1. Tests a late-stage proposal or an existing language feature
2. Is correct
3. Is runnable
4. Is reviewed by someone who can land tests in tests/staging/, not necessarily a test262 maintainer

Requirement for moving a test out of staging is the same as merging tests into the main test262 suite today, e.g. being in the right directory and having the right frontmatter and so forth.

[syg]

Currently all of the tests in staging are related to Temporal. As for why the majority of them fail... perhaps @ptomato can chime in? Those are still supposed to be correct tests, right? (If they're testing for an older iteration of Temporal, for instance, they shouldn't be in there.)

[ptomato]

All of the Temporal tests in staging are correct as far as I know, and at least succeed on the reference Temporal polyfill. They came from what previously was the reference polyfill's own test suite. It's quite likely that they cover things that weren't previously covered in test262 — we committed them to staging so that other implementations could run them while we continue to convert them into a form suitable to land in the main test262 tree. That doesn't exclude that they might contain mistakes, so I'm happy to help debug them the same way as I would with a Temporal test in the main tree.

[linusg]

Thanks all, this is very helpful! I'll took a closer look, and some of the failures are definitely bugs on our end, but a decent chunk of failures is caused by use of the "gregory" calendar in tests.

Given test/staging/Intl402/Temporal/ exists, any Temporal tests relying on Intl should probably be moved there from test/staging/Temporal/.

[linusg]

Same thing with time zones like America/Vancouver where support for them has an effect on the test's outcome.

[ljharb]

Are supporting those timezones not mandated by CLDR, for an Intl-compliant environment?

[linusg]

Sure, hence why I'd like to see separation similar to what already exists with test/intl402/. Baseline Temporal is ISO8601 + UTC with offsets only.

[syg]

Sure, hence why I'd like to see separation similar to what already exists with test/intl402/. Baseline Temporal is ISO8601 + UTC with offsets only.

Hm, probably a good idea to have a intl402-staging/ that's separate. @ptomato WDYT?

[ljharb]

is staging/intl/ or something? or staging/262 and staging/402?

[syg]

is staging/intl/ or something? or staging/262 and staging/402?

Yeah all of those seem fine to me. If the main suite distinguishes 402 vs 262, staging should too.

[ptomato]

See #3680 for this separation. I'll leave this issue open since we still need to document the staging folder.