test262 Testing the Temporal proposal

The Temporal proposal reached Stage 3 on 2021-03-10, and like all ECMA262 proposals, it needs coverage in Test262 to advance to Stage 4. In terms of normative text, it is easily the largest proposal we've seen. Testing it will definitely require significant time, effort, and coordination (even with the proposal's existing tests as a starting point). We'd like to document progress publicly for transparency and to encourage participation.

To get the ball rolling, I'm suggesting that we track progress on an per-API basis using this GitHub.com issue. Compared to alternatives like collaborative document editing services and an in-repository text file, this seems like it balances ease of hosting, maintainability, and proximity to the tests. This approach worked well for ES6's well-known Symbols--the most similar (though much smaller) effort that comes to mind.

Working mode

Anyone can comment here when they'd like to begin writing tests for a specific API. A maintainer will update the list to reflect the ongoing and completed work. When you submit a pull request, please mention this issue (just write "gh-3002") so GitHub displays the status of your work in this thread.

Tests for multiple APIs can be submitted in a single pull request, but patches should be small enough for another contributor to review them in a single session. For instance, it probably makes sense to submit tests for all of the @@toStringTag methods in one patch. There's some subjectivity here, but when in doubt, we should err on the side of smaller patches (and remain open to reviewers' requests to break up pull requests).

Status

[x] Temporal.now.timeZone (reviewed @Ms2ger 2022-04-13)
[x] Temporal.now.instant (reviewed @Ms2ger 2022-04-13)
[ ] Temporal.Now.plainDateTime
[ ] Temporal.Now.plainDateTimeISO
[ ] Temporal.Now.zonedDateTime
[ ] Temporal.Now.zonedDateTimeISO
[ ] Temporal.Now.plainDate
[ ] Temporal.Now.plainDateISO
[ ] Temporal.Now.plainTimeISO
[x] Temporal.PlainDate (reviewed @Ms2ger 2022-06-20)
[x] Temporal.PlainDate.from (reviewed @Ms2ger 2022-06-20)
[x] Temporal.PlainDate.compare (reviewed @Ms2ger 2022-06-20)
[x] Temporal.PlainDate.prototype.constructor (reviewed @Ms2ger 2022-06-20)
[x] Temporal.PlainDate.prototype[ @@toStringTag ] (reviewed @Ms2ger 2022-05-19)
[x] Temporal.PlainDate.prototype.calendar (reviewed @Ms2ger 2022-05-25)
[x] Temporal.PlainDate.prototype.year (reviewed @Ms2ger 2022-05-25)
[x] Temporal.PlainDate.prototype.month (reviewed @Ms2ger 2022-05-25)
[x] Temporal.PlainDate.prototype.monthCode (reviewed @Ms2ger 2022-05-25)
[x] Temporal.PlainDate.prototype.day (reviewed @Ms2ger 2022-05-25)
[x] Temporal.PlainDate.prototype.dayOfWeek (reviewed @Ms2ger 2022-06-20)
[x] Temporal.PlainDate.prototype.dayOfYear (reviewed @Ms2ger 2022-06-20)
[x] Temporal.PlainDate.prototype.weekOfYear (reviewed @Ms2ger 2022-06-20)
[x] Temporal.PlainDate.prototype.daysInWeek (reviewed @Ms2ger 2022-06-20)
[x] Temporal.PlainDate.prototype.daysInMonth (reviewed @Ms2ger 2022-06-20)
[x] Temporal.PlainDate.prototype.daysInYear (reviewed @Ms2ger 2022-06-20)
[x] Temporal.PlainDate.prototype.monthsInYear (reviewed @Ms2ger 2022-06-20)
[x] Temporal.PlainDate.prototype.inLeapYear (reviewed @Ms2ger 2022-06-20)
[x] Temporal.PlainDate.prototype.toPlainYearMonth (reviewed @Ms2ger 2022-06-20)
[x] Temporal.PlainDate.prototype.toPlainMonthDay (reviewed @Ms2ger 2022-06-20)
[x] Temporal.PlainDate.prototype.getISOFields (reviewed @Ms2ger 2022-06-20)
[x] Temporal.PlainDate.prototype.add (reviewed @Ms2ger 2022-06-20)
[x] Temporal.PlainDate.prototype.subtract (reviewed @Ms2ger 2022-06-20)
[x] Temporal.PlainDate.prototype.with (reviewed @Ms2ger 2022-06-20)
[x] Temporal.PlainDate.prototype.withCalendar (reviewed @Ms2ger 2022-06-20)
[x] Temporal.PlainDate.prototype.until (reviewed @Ms2ger 2022-06-20)
[x] Temporal.PlainDate.prototype.since (reviewed @Ms2ger 2022-06-20)
[x] Temporal.PlainDate.prototype.equals (reviewed @Ms2ger 2022-06-20)
[x] Temporal.PlainDate.prototype.toPlainDateTime (reviewed @Ms2ger 2022-06-20)
[x] Temporal.PlainDate.prototype.toZonedDateTime (reviewed @Ms2ger 2022-06-20)
[x] Temporal.PlainDate.prototype.toString (reviewed @Ms2ger 2022-06-20)
[ ] Temporal.PlainDate.prototype.toLocaleString
[x] Temporal.PlainDate.prototype.toJSON (reviewed @Ms2ger 2022-06-20)
[x] Temporal.PlainDate.prototype.valueOf (reviewed @Ms2ger 2022-05-25)
[ ] Temporal.PlainTime
[ ] Temporal.PlainTime.from
[ ] Temporal.PlainTime.compare
[x] Temporal.PlainTime.prototype.constructor (reviewed @ptomato 2022-08-03)
[x] Temporal.PlainTime.prototype[ @@toStringTag ] (reviewed @Ms2ger 2022-05-19)
[x] Temporal.PlainTime.prototype.calendar (reviewed @Ms2ger 2022-05-19)
[x] Temporal.PlainTime.prototype.hour (reviewed @Ms2ger 2022-05-19)
[x] Temporal.PlainTime.prototype.minute (reviewed @Ms2ger 2022-05-19)
[x] Temporal.PlainTime.prototype.second (reviewed @Ms2ger 2022-05-19)
[x] Temporal.PlainTime.prototype.millisecond (reviewed @Ms2ger 2022-05-19)
[x] Temporal.PlainTime.prototype.microsecond (reviewed @Ms2ger 2022-05-19)
[x] Temporal.PlainTime.prototype.nanosecond (reviewed @Ms2ger 2022-05-19)
[ ] Temporal.PlainTime.prototype.add
[ ] Temporal.PlainTime.prototype.subtract
[ ] Temporal.PlainTime.prototype.with
[ ] Temporal.PlainTime.prototype.until
[ ] Temporal.PlainTime.prototype.since
[ ] Temporal.PlainTime.prototype.round
[ ] Temporal.PlainTime.prototype.equals
[ ] Temporal.PlainTime.prototype.toPlainDateTime
[ ] Temporal.PlainTime.prototype.toZonedDateTime
[x] Temporal.PlainTime.prototype.getISOFields (reviewed @ptomato 2022-08-03)
[ ] Temporal.PlainTime.prototype.toString
[ ] Temporal.PlainTime.prototype.toLocaleString
[x] Temporal.PlainTime.prototype.toJSON (reviewed @Ms2ger 2022-05-19)
[x] Temporal.PlainTime.prototype.valueOf (reviewed @Ms2ger 2022-05-19)
[ ] Temporal.PlainDateTime
[ ] Temporal.PlainDateTime.from
[ ] Temporal.PlainDateTime.compare
[x] Temporal.PlainDateTime.prototype.constructor (reviewed @ptomato 2022-08-03)
[x] Temporal.PlainDateTime.prototype[ @@toStringTag ] (reviewed @Ms2ger 2022-05-19)
[ ] Temporal.PlainDateTime.prototype.calendar
[ ] Temporal.PlainDateTime.prototype.year
[ ] Temporal.PlainDateTime.prototype.month
[ ] Temporal.PlainDateTime.prototype.monthCode
[ ] Temporal.PlainDateTime.prototype.day
[ ] Temporal.PlainDateTime.prototype.hour
[ ] Temporal.PlainDateTime.prototype.minute
[ ] Temporal.PlainDateTime.prototype.second
[ ] Temporal.PlainDateTime.prototype.millisecond
[ ] Temporal.PlainDateTime.prototype.microsecond
[ ] Temporal.PlainDateTime.prototype.nanosecond
[ ] Temporal.PlainDateTime.prototype.dayOfWeek
[ ] Temporal.PlainDateTime.prototype.dayOfYear
[ ] Temporal.PlainDateTime.prototype.weekOfYear
[ ] Temporal.PlainDateTime.prototype.daysInWeek
[ ] Temporal.PlainDateTime.prototype.daysInMonth
[ ] Temporal.PlainDateTime.prototype.daysInYear
[ ] Temporal.PlainDateTime.prototype.monthsInYear
[ ] Temporal.PlainDateTime.prototype.inLeapYear
[ ] Temporal.PlainDateTime.prototype.with
[ ] Temporal.PlainDateTime.prototype.withPlainTime
[ ] Temporal.PlainDateTime.prototype.withPlainDate
[ ] Temporal.PlainDateTime.prototype.withCalendar
[ ] Temporal.PlainDateTime.prototype.add
[ ] Temporal.PlainDateTime.prototype.subtract
[ ] Temporal.PlainDateTime.prototype.until
[ ] Temporal.PlainDateTime.prototype.since
[ ] Temporal.PlainDateTime.prototype.round
[ ] Temporal.PlainDateTime.prototype.equals
[ ] Temporal.PlainDateTime.prototype.toString
[ ] Temporal.PlainDateTime.prototype.toLocaleString
[ ] Temporal.PlainDateTime.prototype.toJSON
[ ] Temporal.PlainDateTime.prototype.valueOf
[ ] Temporal.PlainDateTime.prototype.toZonedDateTime
[ ] Temporal.PlainDateTime.prototype.toPlainDate
[ ] Temporal.PlainDateTime.prototype.toPlainYearMonth
[ ] Temporal.PlainDateTime.prototype.toPlainMonthDay
[ ] Temporal.PlainDateTime.prototype.toPlainTime
[x] Temporal.PlainDateTime.prototype.getISOFields (reviewed @ptomato 2022-08-03)
[ ] Temporal.ZonedDateTime
[ ] Temporal.ZonedDateTime.from
[ ] Temporal.ZonedDateTime.compare
[x] Temporal.ZonedDateTime.prototype.constructor (reviewed @ptomato 2022-08-03)
[x] Temporal.ZonedDateTime.prototype[ @@toStringTag ] (reviewed @Ms2ger 2022-05-19)
[ ] Temporal.ZonedDateTime.prototype.calendar
[ ] Temporal.ZonedDateTime.prototype.timeZone
[ ] Temporal.ZonedDateTime.prototype.year
[ ] Temporal.ZonedDateTime.prototype.month
[ ] Temporal.ZonedDateTime.prototype.monthCode
[ ] Temporal.ZonedDateTime.prototype.day
[ ] Temporal.ZonedDateTime.prototype.hour
[ ] Temporal.ZonedDateTime.prototype.minute
[ ] Temporal.ZonedDateTime.prototype.second
[ ] Temporal.ZonedDateTime.prototype.millisecond
[ ] Temporal.ZonedDateTime.prototype.microsecond
[ ] Temporal.ZonedDateTime.prototype.nanosecond
[ ] Temporal.ZonedDateTime.prototype.epochSeconds
[ ] Temporal.ZonedDateTime.prototype.epochMilliseconds
[ ] Temporal.ZonedDateTime.prototype.epochMicroseconds
[ ] Temporal.ZonedDateTime.prototype.epochNanoseconds
[ ] Temporal.ZonedDateTime.prototype.dayOfWeek
[ ] Temporal.ZonedDateTime.prototype.dayOfYear
[ ] Temporal.ZonedDateTime.prototype.weekOfYear
[ ] Temporal.ZonedDateTime.prototype.hoursInDay
[ ] Temporal.ZonedDateTime.prototype.daysInWeek
[ ] Temporal.ZonedDateTime.prototype.daysInMonth
[ ] Temporal.ZonedDateTime.prototype.daysInYear
[ ] Temporal.ZonedDateTime.prototype.monthsInYear
[ ] Temporal.ZonedDateTime.prototype.inLeapYear
[ ] Temporal.ZonedDateTime.prototype.offsetNanoseconds
[ ] Temporal.ZonedDateTime.prototype.offset
[ ] Temporal.ZonedDateTime.prototype.with
[ ] Temporal.ZonedDateTime.prototype.withPlainTime
[ ] Temporal.ZonedDateTime.prototype.withPlainDate
[ ] Temporal.ZonedDateTime.prototype.withTimeZone
[ ] Temporal.ZonedDateTime.prototype.withCalendar
[ ] Temporal.ZonedDateTime.prototype.add
[ ] Temporal.ZonedDateTime.prototype.subtract
[ ] Temporal.ZonedDateTime.prototype.until
[ ] Temporal.ZonedDateTime.prototype.since
[ ] Temporal.ZonedDateTime.prototype.round
[ ] Temporal.ZonedDateTime.prototype.equals
[ ] Temporal.ZonedDateTime.prototype.toString
[ ] Temporal.ZonedDateTime.prototype.toLocaleString
[ ] Temporal.ZonedDateTime.prototype.toJSON
[ ] Temporal.ZonedDateTime.prototype.valueOf
[ ] Temporal.ZonedDateTime.prototype.startOfDay
[ ] Temporal.ZonedDateTime.prototype.toInstant
[ ] Temporal.ZonedDateTime.prototype.toPlainDate
[ ] Temporal.ZonedDateTime.prototype.toPlainTime
[ ] Temporal.ZonedDateTime.prototype.toPlainDateTime
[ ] Temporal.ZonedDateTime.prototype.toPlainYearMonth
[ ] Temporal.ZonedDateTime.prototype.toPlainMonthDay
[ ] Temporal.ZonedDateTime.prototype.getISOFields
[ ] Temporal.Duration
[ ] Temporal.Duration.from
[ ] Temporal.Duration.compare
[x] Temporal.Duration.prototype.constructor (reviewed @ptomato 2022-08-03)
[x] Temporal.Duration.prototype[ @@toStringTag ] (reviewed @Ms2ger 2022-05-19)
[ ] Temporal.Duration.prototype.years
[ ] Temporal.Duration.prototype.months
[ ] Temporal.Duration.prototype.weeks
[ ] Temporal.Duration.prototype.days
[ ] Temporal.Duration.prototype.hours
[ ] Temporal.Duration.prototype.minutes
[ ] Temporal.Duration.prototype.seconds
[ ] Temporal.Duration.prototype.milliseconds
[ ] Temporal.Duration.prototype.microseconds
[ ] Temporal.Duration.prototype.nanoseconds
[ ] Temporal.Duration.prototype.sign
[ ] Temporal.Duration.prototype.blank
[ ] Temporal.Duration.prototype.with
[ ] Temporal.Duration.prototype.negated
[ ] Temporal.Duration.prototype.abs
[ ] Temporal.Duration.prototype.add
[ ] Temporal.Duration.prototype.subtract
[ ] Temporal.Duration.prototype.round
[ ] Temporal.Duration.prototype.total
[ ] Temporal.Duration.prototype.toString
[ ] Temporal.Duration.prototype.toJSON
[ ] Temporal.Duration.prototype.toLocaleString
[ ] Temporal.Duration.prototype.valueOf
[ ] Temporal.Instant
[ ] Temporal.Instant.from
[ ] Temporal.Instant.fromEpochSeconds
[ ] Temporal.Instant.fromEpochMilliseconds
[ ] Temporal.Instant.fromEpochMicroseconds
[ ] Temporal.Instant.fromEpochNanoseconds
[ ] Temporal.Instant.compare
[x] Temporal.Instant.prototype.constructor (reviewed @ptomato 2022-08-03)
[x] Temporal.Instant.prototype[ @@toStringTag ] (reviewed @Ms2ger 2022-05-19)
[ ] Temporal.Instant.prototype.epochSeconds
[ ] Temporal.Instant.prototype.epochMilliseconds
[ ] Temporal.Instant.prototype.epochMicroseconds
[ ] Temporal.Instant.prototype.epochNanoseconds
[ ] Temporal.Instant.prototype.add
[ ] Temporal.Instant.prototype.subtract
[ ] Temporal.Instant.prototype.until
[ ] Temporal.Instant.prototype.since
[ ] Temporal.Instant.prototype.round
[ ] Temporal.Instant.prototype.equals
[ ] Temporal.Instant.prototype.toString
[ ] Temporal.Instant.prototype.toLocaleString
[ ] Temporal.Instant.prototype.toJSON
[ ] Temporal.Instant.prototype.valueOf
[ ] Temporal.Instant.prototype.toZonedDateTime
[ ] Temporal.Instant.prototype.toZonedDateTimeISO
[ ] Temporal.PlainYearMonth
[ ] Temporal.PlainYearMonth.from
[ ] Temporal.PlainYearMonth.compare
[x] Temporal.PlainYearMonth.prototype.constructor (reviewed @ptomato 2022-08-03)
[x] Temporal.PlainYearMonth.prototype[ @@toStringTag ] (reviewed @Ms2ger 2022-05-19)
[ ] Temporal.PlainYearMonth.prototype.calendar
[ ] Temporal.PlainYearMonth.prototype.year
[ ] Temporal.PlainYearMonth.prototype.month
[ ] Temporal.PlainYearMonth.prototype.monthCode
[ ] Temporal.PlainYearMonth.prototype.daysInYear
[ ] Temporal.PlainYearMonth.prototype.daysInMonth
[ ] Temporal.PlainYearMonth.prototype.monthsInYear
[ ] Temporal.PlainYearMonth.prototype.inLeapYear
[ ] Temporal.PlainYearMonth.prototype.with
[ ] Temporal.PlainYearMonth.prototype.add
[ ] Temporal.PlainYearMonth.prototype.subtract
[ ] Temporal.PlainYearMonth.prototype.until
[ ] Temporal.PlainYearMonth.prototype.since
[ ] Temporal.PlainYearMonth.prototype.equals
[ ] Temporal.PlainYearMonth.prototype.toString
[ ] Temporal.PlainYearMonth.prototype.toLocaleString
[ ] Temporal.PlainYearMonth.prototype.toJSON
[ ] Temporal.PlainYearMonth.prototype.valueOf
[ ] Temporal.PlainYearMonth.prototype.toPlainDate
[x] Temporal.PlainYearMonth.prototype.getISOFields (reviewed @ptomato 2022-08-03)
[ ] Temporal.PlainMonthDay
[ ] Temporal.PlainMonthDay.from
[x] Temporal.PlainMonthDay.prototype.constructor (reviewed @ptomato 2022-08-03)
[x] Temporal.PlainMonthDay.prototype[ @@toStringTag ] (reviewed @Ms2ger 2022-05-19)
[ ] Temporal.PlainMonthDay.prototype.calendar
[ ] Temporal.PlainMonthDay.prototype.monthCode
[ ] Temporal.PlainMonthDay.prototype.day
[ ] Temporal.PlainMonthDay.prototype.with
[ ] Temporal.PlainMonthDay.prototype.equals
[ ] Temporal.PlainMonthDay.prototype.toString
[ ] Temporal.PlainMonthDay.prototype.toLocaleString
[ ] Temporal.PlainMonthDay.prototype.toJSON
[ ] Temporal.PlainMonthDay.prototype.valueOf
[ ] Temporal.PlainMonthDay.prototype.toPlainDate
[x] Temporal.PlainMonthDay.prototype.getISOFields (reviewed @ptomato 2022-08-03)
[ ] Temporal.TimeZone
[ ] Temporal.TimeZone.from
[x] Temporal.TimeZone.prototype.constructor (reviewed @ptomato 2022-08-03)
[x] Temporal.TimeZone.prototype[ @@toStringTag ] (reviewed @Ms2ger 2022-05-19)
[ ] Temporal.TimeZone.prototype.id
[ ] Temporal.TimeZone.prototype.getOffsetNanosecondsFor
[ ] Temporal.TimeZone.prototype.getOffsetStringFor
[ ] Temporal.TimeZone.prototype.getPlainDateTimeFor
[ ] Temporal.TimeZone.prototype.getInstantFor
[ ] Temporal.TimeZone.prototype.getPossibleInstantsFor
[ ] Temporal.TimeZone.prototype.getNextTransition
[ ] Temporal.TimeZone.prototype.getPreviousTransition
[ ] Temporal.TimeZone.prototype.toString
[ ] Temporal.TimeZone.prototype.toJSON
[ ] Temporal.Calendar
[ ] Temporal.Calendar.from
[x] Temporal.Calendar.prototype.constructor (reviewed @ptomato 2022-08-03)
[x] Temporal.Calendar.prototype[ @@toStringTag ] (reviewed @Ms2ger 2022-05-19)
[ ] Temporal.Calendar.prototype.id
[ ] Temporal.Calendar.prototype.dateFromFields
[ ] Temporal.Calendar.prototype.yearMonthFromFields
[ ] Temporal.Calendar.prototype.monthDayFromFields
[ ] Temporal.Calendar.prototype.dateAdd
[ ] Temporal.Calendar.prototype.dateUntil
[ ] Temporal.Calendar.prototype.year
[ ] Temporal.Calendar.prototype.month
[ ] Temporal.Calendar.prototype.monthCode
[ ] Temporal.Calendar.prototype.day
[ ] Temporal.Calendar.prototype.dayOfWeek
[ ] Temporal.Calendar.prototype.dayOfYear
[ ] Temporal.Calendar.prototype.weekOfYear
[ ] Temporal.Calendar.prototype.daysInWeek
[ ] Temporal.Calendar.prototype.daysInMonth
[ ] Temporal.Calendar.prototype.daysInYear
[ ] Temporal.Calendar.prototype.monthsInYear
[ ] Temporal.Calendar.prototype.inLeapYear
[ ] Temporal.Calendar.prototype.fields
[ ] Temporal.Calendar.prototype.mergeFields
[ ] Temporal.Calendar.prototype.toString
[ ] Temporal.Calendar.prototype.toJSON
[ ] Temporal.Calendar.prototype.era
[ ] Temporal.Calendar.prototype.eraYear
[ ] Temporal.PlainDate.prototype.era
[ ] Temporal.PlainDate.prototype.eraYear
[ ] Temporal.PlainDateTime.prototype.era
[ ] Temporal.PlainDateTime.prototype.eraYear
[ ] Temporal.PlainYearMonth.prototype.era
[ ] Temporal.PlainYearMonth.prototype.eraYear
[ ] Temporal.ZonedDateTime.prototype.era
[ ] Temporal.ZonedDateTime.prototype.eraYear"

(generated via the following script)

Array.from(document.querySelectorAll('[id^="sec-temporal."], [id^="sec-get-temporal."]'))
  .map((el) => {
    return el
      .querySelector('h1')
      .innerText
      .replace(/^[0-9\. ]*(?:get )?([^ ([]+)(\[[^\]]+])?.*/, '- [ ] $1$2');
  })
  .join('\n');

I executed that in the browser's Developer Console on the proposal's specification text.

Notes about specific things that should have test coverage:

[ ] Behaviour of different rounding modes at a 0.5 boundary, for positive and negative values (from https://github.com/tc39/proposal-temporal/issues/1111)
[ ] Reversibility of ZonedDateTime.p.since/until with DST changes involved (from https://github.com/tc39/proposal-temporal/issues/1140). Given diffUntil = earlier.until(later, { largestUnit: 'days' }) and diffSince = later.since(earlier, { largestUnit: 'days' }), we need to test cases where:
- diffUntil not equal to diffSince
- earlier.add(diffUntil) equals later
- later.subtract(diffUntil) does not equal earlier
- later.subtract(diffSince) equals earlier
- earlier.add(diffSince) does not equal later
- Some of these tests may exist already in Temporal.ZonedDateTime.prototype.since/until, and in staging.
[ ] Add more gregory calendar coverage, possibly via codegen, to check it does the same arithmetic as iso8601 for year > 1 (from https://github.com/tc39/proposal-temporal/issues/1776)
[ ] From tc39/proposal-temporal#1393: Does Temporal check that retrieved methods from user-supplied objects are callable?
[ ] From tc39/proposal-temporal#1393: timeZone.getInstantFor throw for non-Temporal.Instant values
[ ] tc39/proposal-temporal#1047: calendar.fields return includes 'calendar'
[ ] tc39/proposal-temporal#1047: calendar.fields return includes non-calendar fields, e.g. 'nanoseconds'
[ ] tc39/proposal-temporal#1134: Time zones for Temporal.now()
[ ] From tc39/proposal-temporal#1397: Observable lookups & calls for timeZone objects from other temporal methods
[ ] tc39/proposal-temporal#1388: Calendar order of operations (with and)? without Intl
[ ] From tc39/proposal-temporal#1396: PlainMonthDay.prototype.toPlainDate correctly handle [['year', undefined]] for input fields
[ ] From tc39/proposal-temporal#1489: Test CalendarFields calls, including with fields added, and getPossibleInstantsFor type checks
[ ] Check fast-paths from tc39/proposal-temporal#1428:
- [ ] PlainDateTime -> PlainTime
- [ ] ZonedDateTime -> TimeZone
- [ ] PlainTime -> Calendar
- [ ] PlainDate -> relativeTo object
- [ ] Callers of GetOptionalTemporalCalendar and ToTemporalMonthDay, with a not-previously-covered Temporal object (e.g. ToTemporalDate with some object like Object.assign(new Temporal.MonthDay(1, 1), { year: 2000 }))
- [ ] ZonedDateTime -> PlainDate, PlainDateTime, PlainTime
[ ] Check that with() methods don't call getters on Temporal objects
[ ] Check all remaining callers of GetPossibleInstantsFor with checkTimeZonePossibleInstantsIterable
[ ] Check that things of the wrong type in an iterable (calendar.fields return, calendar.fields input, timeZone.getPossibleInstantsFor return) throw before exhausting the iterator
[ ] Check that calendar.fields is always called with the correct list of fields
[ ] Check that Gets occur for every extra field returned by calendar.fields
[ ] Check that toString is not called in CalendarEquals, TimeZoneEquals, ConsolidateCalendars if the two calendars are the same object (tc39/proposal-temporal#1425 point 2)
[ ] Check that any calendar/timezone method that receives an internally-created options object, doesn't cause lookups on Object.prototype (tc39/proposal-temporal#1424)
[ ] Check that property accesses on null-prototype objects created internally in Temporal don't cause lookups on Object.prototype (https://github.com/tc39/proposal-temporal/pull/1875/files#r748524136)
[ ] Add tests for ParseTemporalDurationString overflowing one of the fields to infinity (e.g., PT99999....99999Y)
[ ] Add tests for BalanceDuration causing a Duration field to overflow to infinity and throwing:
- [ ] AddDuration (3x instances)
- [ ] PlainTime.until()/since()
- [ ] PlainDateTime.until()/since()
[ ] Add tests for CreateDurationRecord / CreateTemporalDuration combining mixed signs or overflowing and throwing:
- [ ] AddDateTime
- [ ] new Temporal.Duration()
- [ ] Duration.from()
- [ ] Duration.with()
- [ ] RoundDuration (3rd instance)
[ ] Verify that there is no possible test case that would cause CreateDurationRecord / CreateTemporalDuration to combine values from different sources having different signs:
- [ ] AdjustRoundedDurationDays (last instance)
- [ ] AddDuration (2nd instance)
- [ ] ZonedDateTime.until()/since()
[ ] Verify that there is no possible test case that would cause CreateDurationRecord / CreateTemporalDuration to overflow to infinity:
- [ ] DifferenceZonedDateTime (last instance)
- [ ] RoundDuration
[ ] Check that all (optionName)-undefined.js tests also test plain function objects
[ ] Figure out a sequence of custom calendar and time zone return values that exercise the checks in https://github.com/tc39/proposal-temporal/pull/2387
[ ] Run some DST tests against every time zone (from https://github.com/tc39/proposal-temporal/issues/556)
[ ] Check that a calendar annotation with an ISO date (11111111 or 1111-11-11) as the calendar name throws, and doesn't default to the ISO calendar (from https://github.com/tc39/proposal-temporal/issues/2546#issuecomment-1507724691)
[ ] Check that methods are looked up on calendar and time zone instances even if they are pristine instances of builtin calendars (this can't be tested directly but can be observed e.g. with array iteration on Calendarfields)
[ ] Check when GetOptionsObject is performed relative to the other user-observable operations (currently this is not included in the order-of-operations.js tests) - see tc39/proposal-temporal#2721

Jun 04 '21 17:06 jugglinmike

@ptomato @ms2ger I know this work is near and dear to your hearts :)

Jun 04 '21 17:06 jugglinmike

Thanks for getting this started! FWIW, I've been finding it easier to write tests by topic, rather than by API entrypoint — for example, here are a series of largely identical tests for everything that uses the GetOption abstract operation: https://github.com/tc39/proposal-temporal/pull/1531/files

One thing I'm planning to do is convert the Mocha-style tests of the research polyfill in that repo to test262-style tests, and delete the old-style tests as I go. So anyone picking one of these items can use those old-style tests as a starting point.

Jun 04 '21 17:06 ptomato

Thanks, @jugglinmike! I think that for the moment it would be easiest to keep developing the tests in the proposal repo, so we can run them against the polyfill as a sanity check. Does that make sense to you?

Jun 07 '21 14:06 Ms2ger

Hopefully the polyfill will be finished and deprecated soon tho, so the advantage to keeping the tests there shouldn’t last long?

Jun 07 '21 15:06 ljharb

@Ms2ger I've had luck using Node.js and the test262-harness project to run Test262 against the polyfill:

$ test262-harness \
    --hostType node \
    --hostPath node \
    --prelude inject-temporal.js \
    'test/built-ins/Temporal/**/*'

Where inject-temporal.js looks like this:

const { Temporal } = require('./node_modules/proposal-temporal');

I'm using the version published to npm, but it ought to work for the proposal's repository, too. Does that help at all?

Jun 09 '21 02:06 jugglinmike

@jugglinmike For one thing, we're planning to deprecate that NPM module... An advantage to keeping it in the proposal repository is that we have code coverage metrics for the test262 tests there, and it doesn't require synchronization between git repositories.

(And I can't remember the details off the top of my head, but using require() in the prelude breaks the code coverage metrics, so we generate an IIFE bundle and include that as the prelude.)

Jun 11 '21 17:06 ptomato

https://github.com/tc39/test262/pull/3048

Jul 17 '21 08:07 FrankYFTang

I left one comment in #3049 but it could simplify things in all of these pull requests:

Consider porting over the Temporal asserts from the proposal repo to temporalHelpers.js and using those instead of assert.sameValue("...", ....toJSON()). They should give better error messages and be more specific.

Jul 19 '21 22:07 ptomato

@jugglinmike I noticed that all the getters are missing from this list - here's a change to the code snipped you posted to include them in the list:

-Array.from(document.querySelectorAll('[id^="sec-temporal."]'))
+Array.from(document.querySelectorAll('[id^="sec-temporal."], [id^="sec-get-temporal."]'))
   .map((el) => {
     return el
       .querySelector('h1')
       .innerText
-      .replace(/^[0-9\. ]*([^ ([]+)(\[[^\]]+])?.*/, '- [ ] $1$2');
+      .replace(/^[0-9\. ]*(?:get )?([^ ([]+)(\[[^\]]+])?.*/, '- [ ] $1$2');
   })
   .join('\n');

You also might want to update now => Now :)

Jul 24 '21 14:07 linusg

@jugglinmike I do think it would've been more convenient to develop these in the proposal-temporal repo in order to get code coverage metrics, and then move them over to the test262 repo all at once, but I think that ship has sailed :smile: Just to keep you updated, I discussed it with @Ms2ger and our plan is to finish converting all of the legacy test suite into test262 tests and then move over what we have in the proposal-temporal repo.

Jul 27 '21 23:07 ptomato

@ptomato Mike is on vacation and may not be present in this thread for at least another two weeks.

Aug 06 '21 23:08 rwaldron

I think what @ptomato stated above is perfectly reasonable for Temporal in stage 2 period. Now Temporal is in Stage 3 for about 5 months and in theory the spec is in a shape which is very stable (relative speaking, of course) and would be more benefitial for browser engines under development to sync with the tests right. If that is what the Temporal champion plan to do (move the tests inside the Temporatl repo to test262) then I will not add more Temporal tests PR to test262 until they complete that. Any ETA? To be honest, I am more interest to spend time to change v8 to fulfill the Temporal spec than spend time to add more tests to test262- I only did so because I see no much got added to test262 yet and not aware that Temporal champions plan to do that.

Aug 09 '21 17:08 FrankYFTang

@jugglinmike I do think it would've been more convenient to develop these in the proposal-temporal repo in order to get code coverage metrics, and then move them over to the test262 repo all at once, but I think that ship has sailed :smile: Just to keep you updated, I discussed it with @Ms2ger and our plan is to finish converting all of the legacy test suite into test262 tests and then move over what we have in the proposal-temporal repo.

To come back to this, any comments on the above plan? Since I acknowledge it is taking longer than I'd like, another option would be to copy over all the tests that currently live in tc39/proposal-temporal to tc39/test262 all at once, and at the same time delete them from tc39/proposal-temporal. However, I'd prefer if that happens that we merge it into tc39/test262 as quickly as possible and correct stylistic points afterward, otherwise they will keep getting out of sync between the two repos as we continue converting the legacy test suite into test262 tests. Would a plan like this be preferable?

Sep 15 '21 19:09 ptomato

@jugglinmike Any comments on the above idea to copy over all the tests from tc39/proposal-temporal to tc39/test262 all at once? I can make this PR but I don't want it to get stalled and have the two repos get out of sync in the meantime, so I'd like to have your OK before I do it.

Sep 20 '21 16:09 ptomato

@ptomato I don't understand why you believe that you need to have two sets of the same tests? Here's what I would suggest:

Create a fork of Test262
Add all of the people that will work on your tests as collaborators
Migrate all of your tests to that fork and delete them from the proposal repo.
Use the Test262 fork as the source of truth while developing your tests and your polyfill.
To test your polyfill, use --prelude, as shown here:

# This is executed at the root of the test262 repo
test262-harness --threads=16 --prelude=../proposal-temporal/polyfill/script.js --hostType=jsc --hostPath=`which jsc` 'test/built-ins/Temporal/**/*.js'
FAIL test/built-ins/Temporal/Calendar/prototype/fields/long-input.js (strict mode)
  Expected no error, got RangeError: invalid field name garbage 1

FAIL test/built-ins/Temporal/Calendar/prototype/fields/long-input.js (default)
  Expected no error, got RangeError: invalid field name garbage 1

Ran 830 tests
828 passed
2 failed

The command itself is:

test262-harness --threads=16 --prelude=../proposal-temporal/polyfill/script.js --hostType=jsc --hostPath=`which jsc` 'test/built-ins/Temporal/**/*.js'

This is not a second-class way of working with your tests and Test262: this is exactly what I'm doing for ShadowRealm tests myself (and what I do for all proposals that I work on, which have polyfills before implementations), and also when I verify any Temporal tests that have been contributed so far.

Sep 21 '21 14:09 rwaldron

What's the advantage of that setup over landing the tests in this repository?

Sep 21 '21 14:09 Ms2ger

@Ms2ger Philip is worried about having two sets of tests going out of sync during development. We're worried about a massive deluge of test material that does not have well understood coverage. There are substantial gaps in coverage in the material already contributed; that creates non-trivial auditing work for someone (whether that is a Test262 maintainer, proposal champion, or other contributor, it doesn't matter). When the tests in the fork have sufficient coverage, then we just merge them.

Sep 21 '21 15:09 rwaldron

It's clear that there's no full coverage yet, but it's not clear to me how it's helpful to do the work to review the coverage outside this repo. On the other hand, having them in this repo would be very helpful for the people who are implementing the proposal as we speak.

Sep 21 '21 15:09 Ms2ger

On the other hand, having them in this repo would be very helpful for the people who are implementing the proposal as we speak.

This is actually really compelling. Which engines are currently implementing?

Sep 21 '21 15:09 rwaldron

Hi @rwaldron! I think my concern is actually the opposite — I don't want two sets of tests, only one! We would like to delete the tests from tc39/proposal-temporal as soon as they are present in tc39/test262, and pretty much have a setup prepared just like the one you outlined.

I'm just concerned about the two sets of tests getting out of sync during review, if the PR would (understandably, because large) take a long time to merge, because that would cause a lot of extra work for me. That's why I asked if it could be merged swiftly and stylistic points cleaned up later.

I get your point about coverage, but maybe that could be mitigated using the checklist in the OP of this issue? As we check off each item in the checklist, we could consider the coverage for that item audited. The difference would be that someone working on a checklist item would start from incomplete coverage instead of from zero, which has disadvantages as well as advantages.

The proposal is currently being implemented by V8, SpiderMonkey, and JSC. One other implementation that I'm aware of is in SerenityOS's engine.

Sep 21 '21 16:09 ptomato

Hey there, @ptomato. Let's do it!

@rwaldron and I talked this through today. Although we'd prefer to be more familiar with the test material before merging it, we think the opportunity to assist in-progress implementations, coupled with the correctness (if not completeness) of the tests merged so far, warrant expedition.

Sep 23 '21 23:09 jugglinmike

Thanks, that's great news! I am on vacation tomorrow, so will make this pull request as soon as possible on Monday. If you would like to plan a Matrix chat session or video call next week to go through the test material so I can help you get that familiarity, I would certainly be available for that.

Sep 24 '21 01:09 ptomato

Thanks, Philip, that's very kind!

Regarding coordination: I personally won't be available for reviews until Thursday of next week (that is, September 30). @rwaldron should be around on Monday, but we'll collectively have more bandwidth at the end of the week. You can send a pull request whenever you like, of course--I just know you're interested in limiting the review period.

Sep 25 '21 00:09 jugglinmike

OK, thanks! In that case I won't prioritize the pull request today, but I'll make sure to send it by Wednesday. Does that sound good to you?

Sep 27 '21 16:09 ptomato

Works for me!

Sep 30 '21 16:09 jugglinmike

Are the checkboxes in the OP of this thread still accurate? Or have some additional methods been fully covered by Temporal's 3000+ tests?

Nov 16 '21 02:11 justingrant

I don't think anyone has reviewed the coverage we gained by landing the polyfill tests - I'll see if someone on our side can do it

Nov 16 '21 08:11 Ms2ger

Yes

Sent from Yahoo Mail on Android

On Tue, Nov 16, 2021 at 12:05 AM, @.***> wrote:

I don't think anyone has reviewed the coverage we gained by landing the polyfill tests - I'll see if someone on our side can do it

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

Nov 16 '21 08:11 Bert66cool

Taken care of

Sent from Yahoo Mail on Android

On Tue, Nov 16, 2021 at 12:11 AM, Brett @.***> wrote: Yes

Sent from Yahoo Mail on Android

On Tue, Nov 16, 2021 at 12:05 AM, @.***> wrote:

I don't think anyone has reviewed the coverage we gained by landing the polyfill tests - I'll see if someone on our side can do it

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

Nov 16 '21 11:11 Bert66cool

Will there be tests for each different calendar? Calendars are not symmetric like timezones and it's easy to mistake in only one of them.

Feb 13 '22 07:02 HKalbasi

test262 test262 copied to clipboard

Testing the Temporal proposal

test262
test262 copied to clipboard