icu4x icon indicating copy to clipboard operation
icu4x copied to clipboard

Consider alternative to test runner construct in datetime tests

Open echeran opened this issue 2 years ago • 2 comments

In PR #2874 , updating to the latest CLDR version meant that updates were needed for unit tests' expected values. Doing the update of expected values in test code (& docs) is a reasonable and a good thing because we are providing test coverage for data validity and examples to users.

However, even though the effort is somewhat manual, it was particularity tedious to update the tests in the datetime component. There were a few reasons:

  • locating the source input was difficult. Rust's test error stack trace points to the test invoking the test runner, but easily locating the exact location in JSON of the input data field relied on the existence and uniqueness of the test name string supplied in JSON, and in some cases, the uniqueness of the input & output values from other input / output values
  • the expected/actual (or left/right) strings being compared were printed in Rust string syntax (including Rust style Unicode code point escaping), but the input data is represented in JSON, which uses UTF-16 characters and a different escaping syntax

Even though the test runner is "easy" for the moment when a person authors a large quantity of test input cases, it makes it confusing to maintain and masks some of the benefits of existing Rust test infrastructure, like checking for and enforcing test names, stack traces pointing to the relevant input values when tests are more self-contained, etc.

Some of the benefits of the test runner in the form of saving on boilerplate can be achieved in other ways, such as extracting that boilerplate hiding logic into test helper methods that still take the same concise set of arguments (input and expected values), and converting each test case into an independent Rust unit test that delegates to the appropriate test helper.

echeran avatar Dec 16 '22 19:12 echeran