llrt icon indicating copy to clipboard operation
llrt copied to clipboard
verified publisher icon awslabs

chore(test): Laying the foundations before the full-scale introduction of WPT

Open nabetti1720 opened this issue 8 months ago • 10 comments

Issue # (if available)

related: #968

Description of changes

In this PR, we will be making the following preparations to fully implement WPT:

  • Separate unit tests and wpt directories to enable independent testing. To make it easier to introduce WPT, "make test" now runs only unit tests. If you want to run WPT, use "make test-wpt".
  • For web standard APIs, we recommend using WPT to verify rather than writing unit tests individually, as this is easier and covers most cases. It is best to use unit tests primarily for writing tests for LLRT itself and tests for nodejs compatible modules.
  • There was only one source running the tests, but I split it up for each WebAPI.

The steps are pretty simple:

1. Update sparse-checkout information (only once)
   $ make init-wpt

2. Loading as a submodule (only once)
   $ git submodule add --force -b master https://github.com/web-platform-tests/wpt tests/wpt/wpt

3. Update the local revision (when the remote is updated)
   $ make update-wpt

4. Running WPTs
   $ make test-wpt
result 
% make test-wpt
node build.mjs
Building...

  bundle/js/@llrt/test/worker.js      245.3kb
  bundle/js/stream.js                 194.9kb
  bundle/js/@llrt/test/index.js        20.2kb
  bundle/js/@llrt/test/worker.js.map  411.9kb
  bundle/js/stream.js.map             324.1kb
  bundle/js/@llrt/test/index.js.map    37.3kb

⚡ Done in 51ms
▲ [WARNING] Using direct eval with a bundler is not recommended and may cause problems [direct-eval]

    tests/wpt/resources/idlharness.js:3772:14:
      3772 │         obj = eval(desc);
           ╵               ~~~~

  You can read more about direct eval and bundling here: https://esbuild.github.io/link/direct-eval

▲ [WARNING] Comparison with -0 using the "===" operator will also match 0 [equals-negative-zero]

    tests/wpt/resources/testharness.js:1506:22:
      1506 │           if (val === -0 && 1 / val === -Infinity) {
           ╵                       ~~

  Floating-point equality is defined such that 0 and -0 are equal, so "x === -0" returns true for
  both 0 and -0. You need to use "Object.is(x, -0)" instead to test for -0.

2 warnings

  bundle/js/__tests__/wpt/url.test.js                            560.4kb
  bundle/js/__tests__/wpt/streams.piping.test.js                 287.9kb
  bundle/js/__tests__/wpt/streams.readable-byte-streams.test.js  287.8kb
  bundle/js/__tests__/wpt/streams.readable-streams.test.js       287.7kb
  bundle/js/__tests__/wpt/streams.writable-streams.test.js       287.7kb
  bundle/js/__tests__/wpt/fetch.api.request.test.js              269.4kb
  bundle/js/__tests__/wpt/fetch.api.basic.test.js                269.2kb
  bundle/js/__tests__/wpt/fetch.api.response.test.js             268.8kb
  ...and 104 more output files...

⚡ Done in 113ms
npx pretty-quick --pattern "tests/wpt/**/*.{js,ts,json}"
🔍  Finding changed files since git revision null.
🎯  Found 0 changed files.
✅  Everything is awesome!
cargo run -- test -d bundle/js/__tests__/wpt
   Compiling llrt_fetch v0.5.1-beta (/Users/shinya/Workspaces/llrt/modules/llrt_fetch)
   Compiling llrt_core v0.5.1-beta (/Users/shinya/Workspaces/llrt/llrt_core)
   Compiling llrt_modules v0.5.1-beta (/Users/shinya/Workspaces/llrt/llrt_modules)
   Compiling llrt v0.5.1-beta (/Users/shinya/Workspaces/llrt/llrt)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 12.09s
     Running `target/debug/llrt test -d bundle/js/__tests__/wpt`
✘ FileAPI.blob.test.js
✘ FileAPI.file.test.js
✔ WebCryptoAPI.derive_bits_keys.test.js
✘ WebCryptoAPI.test.js
✘ console.test.js
✘ fetch.api.abort.test.js
✘ fetch.api.basic.test.js
✘ fetch.api.body.test.js
✘ fetch.api.headers.test.js
✘ fetch.api.request.test.js
✘ fetch.api.response.test.js
✘ WebCryptoAPI.digest.test.js
✘ encoding.test.js
✘ streams.readable-byte-streams.test.js
✘ streams.readable-streams.test.js
✘ streams.writable-streams.test.js
✘ webidl.ecmascript-binding.es-exceptions.test.js
✘ url.test.js
✘ hr-time.test.js
⠧ Testing [======================--]74/181: streams.piping.test.js
 FAIL  FileAPI.blob.test.js -70.692ms
FileAPI/blob 0.000ms
  ✘ should pass Blob-array-buffer.any.js tests 5.891ms
Error: [Blob.arrayBuffer()] promise_test: Unhandled rejection with value: object "ReferenceError: assert_true is not defined"

  ✘ should pass Blob-bytes.any.js tests 10.455ms
Error: [Blob.bytes()] promise_test: Unhandled rejection with value: object "ReferenceError: assert_true is not defined"

  ✘ should pass Blob-slice-overflow.any.js tests 4936.564ms
Error: Test timed out after 5000ms
    at tick (llrt:test/index:480:15)
    at <anonymous> (llrt:test/index:160:12)


 FAIL  FileAPI.file.test.js 17.850ms
FileAPI/file 15.990ms
  ✘ should pass File-constructor.any.js tests 13.379ms
Error: [Empty File fileBits] assert_equals: expected 0 but got 13


 PASS  WebCryptoAPI.derive_bits_keys.test.js 1.977ms
WebCryptoAPI/derive_bits_keys 0.811ms

 FAIL  WebCryptoAPI.digest.test.js 373.611ms
WebCryptoAPI/digest 370.257ms
  ✘ should pass digest.https.any.js tests 368.467ms
Error: [SHA-1 with short source data and altered buffer after call] assert_true: digest() yielded expected result for sha-1:short expected true got false


 FAIL  WebCryptoAPI.test.js 77.189ms
WebCryptoAPI 75.629ms
  ✘ should pass crypto_key_cached_slots.https.any.js tests 4.024ms
Error: [CryptoKey.algorithm getter returns cached object] assert_true: expected true got false

  ✘ should pass historical.any.js tests 49.288ms
Error: [Non-secure context window does not have access to crypto.subtle] assert_equals: expected (undefined) undefined but got (object) object "[object Object]"

  ✔ should pass randomUUID.https.any.js tests 18.840ms

 FAIL  console.test.js 722.697ms
console 721.556ms
  ✘ should pass console-is-a-namespace.any.js tests 6.642ms
Error: [console has the right property descriptors] assert_equals: must not be enumerable expected false but got true

  ✘ should pass console-label-conversion.any.js tests 12.972ms
Error: [console.count()'s label gets converted to string via label.toString() when label is an object] not a function

  ✔ should pass console-log-large-array.any.js tests 692.013ms
  ✘ should pass console-namespace-object-class-string.any.js tests 4.355ms
Error: [@@toStringTag exists on the namespace object with the appropriate descriptor] assert_own_property: expected property symbol "Symbol(Symbol.toStringTag)" missing

  ✔ should pass console-tests-historical.any.js tests 2.119ms

 FAIL  encoding.test.js 1301.800ms
encoding 1300.551ms
  ✔ should pass api-basics.any.js tests 6.275ms
  ✔ should pass api-invalid-label.any.js tests 357.616ms
  ✔ should pass api-replacement-encodings.any.js tests 3.189ms
  ✘ should pass api-surrogates-utf8.any.js tests 3.138ms
Error: [Invalid surrogates encoded into UTF-8: Surrogate half (low)] Conversion from string failed: invalid utf-8 sequence of 1 bytes from index 0

  ✘ should pass encodeInto.any.js tests 21.625ms
Error: [encodeInto() into ArrayBuffer with U+d834AU+df06A¥Hi and destination length 10, offset 0, filler 0] Conversion from string failed: invalid utf-8 sequence of 1 bytes from index 0

  ✘ should pass textdecoder-arguments.any.js tests 7.667ms
Error: [TextDecoder decode() with explicit undefined] assert_equals: Undefined as first arg should flush the stream expected "\ufffd" but got ""

  ✘ should pass textdecoder-byte-order-marks.any.js tests 9.823ms
Error: [Byte-order marks: utf-8] assert_not_equals: Mismatching BOM should not be ignored - treated as garbage bytes. got disallowed value "z¢水𝄞􏿽"

  ✘ should pass textdecoder-copy.any.js tests 8.492ms
Error: [Modify buffer after passing it in (ArrayBuffer)] assert_equals: expected "" but got "\ufffd"

  ✘ should pass textdecoder-eof.any.js tests 4.435ms
Error: [TextDecoder end-of-queue handling using stream: true] Error calling function with 0 argument(s) while 1 where expected

  ✘ should pass textdecoder-fatal-single-byte.any.js tests 823.793ms
Error: [Not throw: IBM866 has a pointer 0] The "IBM866" encoding is not supported

  ✘ should pass textdecoder-fatal-streaming.any.js tests 3.375ms
Error: [Fatal flag, non-streaming cases] assert_equals: Unterminated UTF-8 sequence should emit replacement character if fatal flag is unset expected "\ufffd" but got ""

  ✘ should pass textdecoder-fatal.any.js tests 9.746ms
Error: [Error seen with fatal does not prevent future decodes] assert_throws_js: decode() should throw on incomplete sequence function "() => decoder.decode(new DataView(bytes.buffer, 0, 2))" did not throw

  ✔ should pass textdecoder-ignorebom.any.js tests 2.562ms
  ✔ should pass textdecoder-labels.any.js tests 12.379ms
  ✘ should pass textdecoder-streaming.any.js tests 10.159ms
Error: [Streaming decode: utf-8, 1 byte window (ArrayBuffer)] Error calling function with 0 argument(s) while 1 where expected

  ✔ should pass textdecoder-utf16-surrogates.any.js tests 2.916ms
  ✔ should pass textencoder-constructor-non-utf.any.js tests 2.418ms
  ✘ should pass textencoder-utf16-surrogates.any.js tests 2.367ms
Error: [USVString handling: lone surrogate lead] Conversion from string failed: invalid utf-8 sequence of 1 bytes from index 0


 FAIL  fetch.api.abort.test.js 9.975ms
fetch/api/abort 8.645ms
  ✘ should pass cache.https.any.js tests 5.593ms
Error: [Signals are not stored in the cache API] promise_test: Unhandled rejection with value: object "ReferenceError: caches is not defined"


 FAIL  fetch.api.basic.test.js 5262.799ms
fetch/api/basic 5262.022ms
  ✘ should pass accept-header.any.js tests 7.413ms
Error: [Request through fetch should have 'accept' header with value '*/*'] promise_test: Unhandled rejection with value: object "Error: invalid format"

  ✘ should pass conditional-get.any.js tests 5.100ms
Error: [Testing conditional GET with ETags] token is not defined

  ✘ should pass error-after-response.any.js tests 6.934ms
Error: [Response reader read() promise should reject after a network error happening after resolving fetch promise] promise_test: Unhandled rejection with value: object "Error: invalid format"

  ✘ should pass gc.any.js tests 6.770ms
Error: [GC/CC should not abruptly close the stream while being consumed by Response] assert_equals: The buffer should be 5-byte long expected 5 but got 15

  ✘ should pass header-value-combining.any.js tests 17.132ms
Error: [response.headers.get('content-length') expects 0] promise_test: Unhandled rejection with value: object "Error: invalid format"

  ✘ should pass header-value-null-byte.any.js tests 4.741ms
Error: [Ensure fetch() rejects null bytes in headers] promise_rejects_js: function "function() {
            throw e;
          }" threw object "Error: invalid format" ("Error") expected instance of function "function TypeError() {
    [native code]
}" ("TypeError")

  ✔ should pass historical.any.js tests 6.766ms
  ✘ should pass http-response-code.any.js tests 3.193ms
Error: [Fetch on 425 response should not be retried for non TLS early data.] promise_test: Unhandled rejection with value: object "ReferenceError: token is not defined"

  ✘ should pass mode-no-cors.sub.any.js tests 6.340ms
Error: [Fetch top.txt with no-cors mode] promise_test: Unhandled rejection with value: object "Error: invalid format"

  ✘ should pass request-head.any.js tests 6.415ms
Error: [Fetch with HEAD with body] promise_rejects_js: function "function() {
            throw e;
          }" threw object "Error: client error (UserAbsoluteUriRequired)" ("Error") expected instance of function "function TypeError() {
    [native code]
}" ("TypeError")

  ✘ should pass request-headers-case.any.js tests 3.967ms
Error: [Multiple headers with the same name, different case (THIS-is-A-test first)] promise_test: Unhandled rejection with value: object "Error: client error (UserAbsoluteUriRequired)"

  ✘ should pass request-headers-nonascii.any.js tests 4.720ms
Error: [Non-ascii bytes in request headers] promise_test: Unhandled rejection with value: object "Error: invalid format"

  ✘ should pass request-headers.any.js tests 16.416ms
Error: [Fetch with GET] promise_test: Unhandled rejection with value: object "Error: invalid format"

  ✘ should pass request-upload.any.js tests 37.244ms
Error: [Fetch with PUT with body] promise_test: Unhandled rejection with value: object "Error: client error (UserAbsoluteUriRequired)"

  ✘ should pass request-upload.h2.any.js tests 5002.718ms
Error: Timeout after 5000ms

  ✘ should pass response-null-body.any.js tests 14.252ms
Error: [Response.body is null for responses with status=204 (method=GET)] promise_test: Unhandled rejection with value: object "Error: invalid format"

  ✘ should pass response-url.sub.any.js tests 6.034ms
Error: [Testing response url getter with http://{{host}}:{{ports[http][0]}}/ada] promise_test: Unhandled rejection with value: object "Error: invalid uri character"

  ✘ should pass scheme-about.any.js tests 14.222ms
Error: [Fetching about:blank with method GET is KO] promise_rejects_js: function "function() {
            throw e;
          }" threw object "Error: client error (UserAbsoluteUriRequired)" ("Error") expected instance of function "function TypeError() {
    [native code]
}" ("TypeError")

  ✘ should pass scheme-data.any.js tests 10.790ms
Error: [Fetching data:,response%27s%20body is OK] assert_equals: Response's body is correct expected "response's body" but got "response%27s%20body"

  ✘ should pass scheme-others.sub.any.js tests 21.520ms
Error: [Fetching aaa://{{host}}:{{ports[http][0]}}/ is KO] promise_rejects_js: function "function() {
            throw e;
          }" threw object "Error: invalid uri character" ("Error") expected instance of function "function TypeError() {
    [native code]
}" ("TypeError")

  ✘ should pass status.h2.any.js tests 4.727ms
Error: [statusText over H2 for status 200 should be the empty string] promise_test: Unhandled rejection with value: object "Error: client error (UserAbsoluteUriRequired)"

  ✘ should pass stream-response.any.js tests 2.932ms
Error: [Stream response's body when content-type is present] promise_test: Unhandled rejection with value: object "Error: invalid format"

  ✘ should pass stream-safe-creation.any.js tests 16.416ms
Error: [throwing Object.prototype.start accessor should not affect stream creation by 'fetch'] promise_test: Unhandled rejection with value: object "Error: client error (UserAbsoluteUriRequired)"

  ✘ should pass text-utf8.any.js tests 16.258ms
Error: [UTF-8 with BOM with Request.text()] assert_equals: Request.text() should decode data as UTF-8 expected "三村かな子" but got "三村かな子"


 FAIL  fetch.api.body.test.js 42.983ms
fetch/api/body 41.289ms
  ✘ should pass formdata.any.js tests 5.281ms
Error: [Consume empty response.formData() as FormData] promise_test: Unhandled rejection with value: object "ReferenceError: FormData is not defined"

  ✘ should pass mime-type.any.js tests 33.545ms
Error: [Request: overriding explicit Content-Type] assert_equals: expected "test/test" but got ""


 FAIL  fetch.api.headers.test.js 141.036ms
fetch/api/headers 139.717ms
  ✘ should pass header-setcookie.any.js tests 30.710ms
Error: [Headers iterator does not special case set-cookie2 headers] assert_equals: Array length is not equal expected 1 but got 2

  ✘ should pass headers-basic.any.js tests 38.450ms
Error: [Create headers with null should throw] assert_throws_js: function "function() { new Headers(parameter) }" did not throw

  ✔ should pass headers-casing.any.js tests 5.369ms
  ✘ should pass headers-combine.any.js tests 10.872ms
Error: [Create headers using same name for different values] assert_equals: expected "doubleValue1, doubleValue2" but got "doubleValue1"

  ✘ should pass headers-errors.any.js tests 8.561ms
Error: [Create headers giving an array having three strings as init argument] assert_throws_js: function "function() { new Headers([["invalid", "invalidValue1", "invalidValue2"]]); }" did not throw

  ✘ should pass headers-no-cors.any.js tests 13.999ms
Error: [Loading data…] promise_test: Unhandled rejection with value: object "Error: invalid format"

  ✘ should pass headers-normalize.any.js tests 7.968ms
Error: [Create headers with not normalized values] assert_equals: name: name1 has normalized value: space expected "space" but got " space "

  ✘ should pass headers-record.any.js tests 10.592ms
Error: [Passing null to Headers constructor] assert_throws_js: function "function() {
    var h = new Headers(null);
  }" did not throw

  ✔ should pass headers-structure.any.js tests 5.883ms

 FAIL  fetch.api.request.test.js 286.133ms
fetch/api/request 284.020ms
  ✘ should pass forbidden-method.any.js tests 5.905ms
Error: [Request() with a forbidden method CONNECT must throw.] assert_throws_js: function "function() { new Request('./', {method: method}); }" did not throw

  ✘ should pass request-bad-port.any.js tests 160.001ms
Error: [Request on bad port 1 should throw TypeError.] promise_rejects_js: function "function() {
            throw e;
          }" threw object "Error: client error (UserAbsoluteUriRequired)" ("Error") expected instance of function "function TypeError() {
    [native code]
}" ("TypeError")

  ✔ should pass request-constructor-init-body-override.any.js tests 2.856ms
  ✘ should pass request-consume.any.js tests 23.081ms
Error: [Consume String request's body as text] assert_false: bodyUsed is false at init expected false got true

  ✘ should pass request-disturbed.any.js tests 6.049ms
Error: [Request's body: initial state] assert_true: non-null body type expected true got false

  ✘ should pass request-headers.any.js tests 38.077ms
Error: [Adding invalid request header "Accept-Charset: KO"] assert_equals: expected (object) null but got (string) "KO"

  ✘ should pass request-init-contenttype.any.js tests 7.296ms
Error: [Default Content-Type for Request with Blob body (set type)] assert_equals: expected (string) "a/b; c=d" but got (object) null

  ✘ should pass request-init-priority.any.js tests 9.381ms
Error: [new Request() throws a TypeError if any of RequestInit's members' values are invalid] assert_throws_js: a new Request() must throw a TypeError if RequestInit's priority is an invalid value function "() => {
    new Request("", {priority: 'invalid'});
  }" did not throw

  ✘ should pass request-init-stream.any.js tests 8.719ms
Error: [Constructing a Request with a stream on which getReader() is called] assert_throws_js: function "() => new Request("...", { method, body, duplex })" did not throw

  ✘ should pass request-keepalive.any.js tests 4.819ms
Error: [keepalive flag] assert_false: default expected false got true

  ✘ should pass request-structure.any.js tests 10.625ms
Error: [Request has formData method] assert_true: request has formData method expected true got false


 FAIL  fetch.api.response.test.js 5159.782ms
fetch/api/response 5158.911ms
  ✘ should pass json.any.js tests 3.150ms
Error: [Ensure the correct JSON parser is used] promise_test: Unhandled rejection with value: object "Error: "{ "b": 1, "a": 2, "b": 3 }" not valid JSON at index 0 ('ï')"

  ✘ should pass response-blob-realm.any.js tests 5002.431ms
Error: Timeout after 5000ms

  ✘ should pass response-cancel-stream.any.js tests 15.617ms
Error: [Cancelling a starting blob Response stream] cannot read property 'cancel' of null

  ✘ should pass response-clone.any.js tests 16.156ms
Error: [Check Response's clone with default values, without body] assert_equals: Expect default response.type is default expected "default" but got "basic"

  ✘ should pass response-error-from-stream.any.js tests 12.128ms
Error: [ReadableStream start() Error propagates to Response.arrayBuffer() Promise] assert_unreached: Should have rejected: CustomTestError should propagate Reached unreachable code

  ✘ should pass response-error.any.js tests 6.702ms
Error: [Throws RangeError when responseInit's status is 0] assert_throws_js: Expect RangeError exception when status is 0 function "function() { new Response("", { "status" : status }); }" did not throw

  ✘ should pass response-from-stream.any.js tests 4.067ms
Error: [Constructing a Response with a stream on which getReader() is called] assert_throws_js: function "() => new Response(stream)" did not throw

  ✘ should pass response-headers-guard.any.js tests 2.852ms
Error: [Ensure response headers are immutable] promise_test: Unhandled rejection with value: object "Error: invalid format"

  ✘ should pass response-init-001.any.js tests 5.519ms
Error: [Check default value for type attribute] assert_equals: Expect default response.type is default expected "default" but got "basic"

  ✘ should pass response-init-contenttype.any.js tests 7.050ms
Error: [Default Content-Type for Response with Blob body (set type)] assert_equals: expected (string) "a/b; c=d" but got (object) null

  ✘ should pass response-static-error.any.js tests 2.408ms
Error: [the 'guard' of the Headers instance should be immutable] assert_throws_js: function "function () { headers.append('name', 'value'); }" did not throw

  ✘ should pass response-static-json.any.js tests 11.868ms
Error: [Check response returned by static json() with init undefined] promise_test: Unhandled rejection with value: object "TypeError: Error converting from js 'undefined' into type 'object'"

  ✘ should pass response-static-redirect.any.js tests 9.518ms
Error: [Check default redirect response] assert_equals: expected "default" but got "basic"

  ✘ should pass response-stream-bad-chunk.any.js tests 5.152ms
Error: [ReadableStream with non-Uint8Array chunk passed to Response.arrayBuffer() causes TypeError] assert_unreached: Should have rejected: TypeError should propagate Reached unreachable code

  ✘ should pass response-stream-disturbed-1.any.js tests 6.581ms
Error: [Getting blob after getting the Response body - not disturbed, not locked (body source: fetch)] promise_test: Unhandled rejection with value: object "ReferenceError: responseFromBodySource is not defined"

  ✘ should pass response-stream-disturbed-2.any.js tests 6.375ms
Error: [Getting blob after getting a locked Response body (body source: fetch)] promise_test: Unhandled rejection with value: object "ReferenceError: responseFromBodySource is not defined"

  ✘ should pass response-stream-disturbed-3.any.js tests 6.387ms
Error: [Getting blob after reading the Response body (body source: fetch)] promise_test: Unhandled rejection with value: object "ReferenceError: responseFromBodySource is not defined"

  ✘ should pass response-stream-disturbed-4.any.js tests 6.460ms
Error: [Getting blob after cancelling the Response body (body source: fetch)] promise_test: Unhandled rejection with value: object "ReferenceError: responseFromBodySource is not defined"

  ✘ should pass response-stream-disturbed-5.any.js tests 6.347ms
Error: [Getting a body reader after consuming as blob (body source: fetch)] promise_test: Unhandled rejection with value: object "ReferenceError: responseFromBodySource is not defined"

  ✘ should pass response-stream-disturbed-6.any.js tests 3.721ms
Error: [A non-closed stream on which read() has been called] assert_true: After calling stream.read() expected true got false

  ✘ should pass response-stream-disturbed-by-pipe.any.js tests 2.016ms
Error: [using pipeTo on Response body should disturb it synchronously] cannot read property 'pipeTo' of null

  ✘ should pass response-stream-with-broken-then.any.js tests 7.127ms
Error: [Attempt to inject {done: false, value: bye} via Object.prototype.then.] assert_equals: The value should be "hello". expected "hello" but got "[object Object]"


 FAIL  hr-time.test.js 2008.768ms
hr-time 2007.951ms
  ✘ should pass basic.any.js tests 2003.335ms
Error: [Performance interface extends EventTarget.] not a function

  ✔ should pass monotonic-clock.any.js tests 2.809ms

 PASS  streams.piping.test.js 531.031ms
streams/piping 530.197ms
  ✔ should pass abort.any.js tests 30.155ms
  ✔ should pass close-propagation-backward.any.js tests 12.451ms
  ✔ should pass close-propagation-forward.any.js tests 62.490ms
  ✔ should pass error-propagation-backward.any.js tests 105.896ms
  ✔ should pass error-propagation-forward.any.js tests 239.463ms
  ✔ should pass flow-control.any.js tests 10.620ms
  ✔ should pass general.any.js tests 33.507ms
  ✔ should pass multiple-propagation.any.js tests 12.171ms
  ✔ should pass pipe-through.any.js tests 15.626ms
  ✔ should pass then-interception.any.js tests 3.401ms

 FAIL  streams.readable-byte-streams.test.js 2192.982ms
streams/readable-byte-streams 2192.160ms
  ✔ should pass bad-buffers-and-views.any.js tests 10.082ms
  ✔ should pass construct-byob-request.any.js tests 4.041ms
  ✔ should pass enqueue-with-detached-buffer.any.js tests 2.633ms
  ✔ should pass general.any.js tests 58.043ms
  ✘ should pass patched-global.any.js tests 31.414ms
Error: [Patched then() sees byobRequest after filling all pending pull-into descriptors] Error borrowing class: can't borrow a value as it is already borrowed

  ✔ should pass read-min.any.js tests 16.902ms
  ✔ should pass respond-after-enqueue.any.js tests 4.044ms
  ✔ should pass tee.any.js tests 40.529ms
  ✔ should pass templated.any.js tests 2019.294ms

 FAIL  streams.readable-streams.test.js 1573.460ms
streams/readable-streams 1572.623ms
  ✔ should pass async-iterator.any.js tests 22.297ms
  ✔ should pass bad-strategies.any.js tests 4.964ms
  ✔ should pass bad-underlying-sources.any.js tests 8.552ms
  ✔ should pass cancel.any.js tests 45.429ms
  ✔ should pass constructor.any.js tests 2.087ms
  ✔ should pass count-queuing-strategy-integration.any.js tests 4.604ms
  ✔ should pass default-reader.any.js tests 11.177ms
  ✔ should pass floating-point-total-queue-size.any.js tests 3.500ms
  ✘ should pass from.any.js tests 19.617ms
Error: [ReadableStream.from: cancel() rejects when return() is not a method] promise_test: Unhandled rejection with value: object "TypeError: Error converting from js 'int' into type 'function'"

  ✔ should pass garbage-collection.any.js tests 221.982ms
  ✔ should pass general.any.js tests 162.154ms
  ✘ should pass owning-type-message-port.any.js tests 2.683ms
Error: [Transferred MessageChannel works as expected] promise_test: Unhandled rejection with value: object "ReferenceError: MessageChannel is not defined"

  ✘ should pass owning-type.any.js tests 3.193ms
Error: [ReadableStream can be constructed with owning type] Error converting from js 'string' into type 'ReadableStreamType'

  ✘ should pass patched-global.any.js tests 3.497ms
Error: [tee() should not call Promise.prototype.then()] patched then() called

  ✔ should pass reentrant-strategies.any.js tests 6.706ms
  ✔ should pass tee.any.js tests 17.521ms
  ✔ should pass templated.any.js tests 1026.700ms

 FAIL  streams.writable-streams.test.js 355.050ms
streams/writable-streams 354.523ms
  ✔ should pass aborting.any.js tests 38.068ms
  ✔ should pass bad-strategies.any.js tests 4.170ms
  ✔ should pass bad-underlying-sinks.any.js tests 6.291ms
  ✔ should pass byte-length-queuing-strategy.any.js tests 204.489ms
  ✔ should pass close.any.js tests 13.077ms
  ✔ should pass constructor.any.js tests 4.961ms
  ✔ should pass count-queuing-strategy.any.js tests 3.333ms
  ✔ should pass error.any.js tests 3.276ms
  ✔ should pass floating-point-total-queue-size.any.js tests 3.249ms
  ✔ should pass garbage-collection.any.js tests 13.579ms
  ✘ should pass general.any.js tests 6.915ms
Error: [WritableStream's strategy.size should not be called as a method] promise_test: Unhandled rejection with value: object "Error: size called as a method"

  ✔ should pass properties.any.js tests 4.039ms
  ✔ should pass reentrant-strategy.any.js tests 5.631ms
  ✔ should pass start.any.js tests 4.864ms
  ✔ should pass write.any.js tests 33.910ms

 FAIL  url.test.js 393.123ms
url 392.561ms
  ✘ should pass url-constructor.any.js tests 170.396ms
Error: [Parsing: <file:///w|/m> without base] assert_equals: href expected "file:///w:/m" but got "file:///w|/m"

  ✘ should pass url-origin.any.js tests 43.814ms
Error: [Origin parsing: <blob:blob:https://example.org/> without base] assert_equals: origin expected "null" but got "https://example.org"

  ✔ should pass url-searchparams.any.js tests 3.094ms
  ✘ should pass url-setters-stripping.any.js tests 43.567ms
Error: [Setting port with leading U+0000 (https:)] assert_equals: property expected "8000" but got ""

  ✘ should pass url-setters.any.js tests 58.268ms
Error: [URL: Setting <http://example.net:8080>.host = 'example.com:invalid' Anything other than ASCII digit stops the port parser in a setter but is not an error] assert_equals: expected "http://example.com:8080/" but got "http://example.com/"

  ✔ should pass url-statics-canparse.any.js tests 2.949ms
  ✔ should pass url-statics-parse.any.js tests 2.679ms
  ✔ should pass url-tojson.any.js tests 1.695ms
  ✘ should pass urlencoded-parser.any.js tests 17.428ms
Error: [request.formData() with input: test] not a function

  ✔ should pass urlsearchparams-append.any.js tests 2.464ms
  ✘ should pass urlsearchparams-constructor.any.js tests 6.679ms
Error: [URLSearchParams constructor, FormData.] FormData is not defined

  ✘ should pass urlsearchparams-delete.any.js tests 4.514ms
Error: [Changing the query of a URL with an opaque path with trailing spaces] assert_equals: expected "space   %20" but got "space"

  ✔ should pass urlsearchparams-foreach.any.js tests 2.812ms
  ✔ should pass urlsearchparams-get.any.js tests 2.037ms
  ✔ should pass urlsearchparams-getall.any.js tests 2.211ms
  ✔ should pass urlsearchparams-has.any.js tests 2.371ms
  ✔ should pass urlsearchparams-set.any.js tests 1.881ms
  ✔ should pass urlsearchparams-size.any.js tests 2.050ms
  ✘ should pass urlsearchparams-sort.any.js tests 10.698ms
Error: [Parse and sort: ffi&🌈] assert_array_equals: expected property 0 to be "🌈" but got "ffi" (expected array ["🌈", ""] got ["ffi", ""])

  ✘ should pass urlsearchparams-stringifier.any.js tests 4.695ms
Error: [URLSearchParams connected to URL] assert_equals: expected "http://www.example.com/?a=b%2Cc&x=y" but got "http://www.example.com/?a=b,c&x=y"


 FAIL  webidl.ecmascript-binding.es-exceptions.test.js 24.640ms
webidl/ecmascript-binding/es-exceptions 24.105ms
  ✔ should pass DOMException-constants.any.js tests 7.411ms
  ✔ should pass DOMException-constructor-and-prototype.any.js tests 2.150ms
  ✔ should pass DOMException-constructor-behavior.any.js tests 6.811ms
  ✘ should pass DOMException-custom-bindings.any.js tests 4.105ms
Error: [Cannot construct without new] assert_throws_js: function "() => DOMException()" did not throw

  ✘ should pass DOMException-is-error.any.js tests 1.924ms
Error: [Untitled] assert_true: expected true got false


 ✔ TESTS FAILED  5414.818ms
74 passed, 104 failed, 0 skipped, 181 tests

...The re-output part of the results is omitted...

make: *** [test-wpt] Error 1

Checklist

  • [x] Created unit tests in tests/unit and/or in Rust for my feature if needed
  • [x] Ran make fix to format JS and apply Clippy auto fixes
  • [x] Made sure my code didn't add any additional warnings: make check
  • [x] Added relevant type info in types/ directory
  • [x] Updated documentation if needed (API.md/README.md/Other)

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

nabetti1720 avatar May 10 '25 07:05 nabetti1720

I'm skipping the test for now, but if you test writable-streams in parallel, the following code will cause a panic:

https://github.com/awslabs/llrt/blob/300743ec1abd0133b6e44536d13a7dde0b8221a9/libs/llrt_utils/src/primordials.rs#L46-L64

thread 'main' panicked at /Users/shinya/Workspaces/llrt/libs/llrt_utils/src/primordials.rs:58:35:
called `Option::unwrap()` on a `None` value
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

nabetti1720 avatar May 10 '25 09:05 nabetti1720

The panic looks like a bug to me. All modules should work in parallel. We will have to investigate.

Sytten avatar May 10 '25 13:05 Sytten

I started working on some WebCrypto WPTs in https://github.com/awslabs/llrt/tree/feat/wpt-webcrypto . Issue is, that some WTP test suites are very browser oriented (and requires code generation via a python script). I like the idea that this PR takes a holistic approach for WPTs.

richarddavison avatar May 11 '25 19:05 richarddavison

@richarddavison I would be very grateful if you could fix the bug in WebCryptoAPI. However, if the test suite is merged together, it will cause a conflict, so if possible, it would be helpful if you could revert the changes to the test suite before merging.

Also, I apologize for leaving this in draft status for so long. I would like to introduce git submodule/sparse-checkout to WPT and work on making it easier to update test suite, so please wait a little longer.

nabetti1720 avatar May 11 '25 23:05 nabetti1720

@richarddavison I would be very grateful if you could fix the bug in WebCryptoAPI. However, if the test suite is merged together, it will cause a conflict, so if possible, it would be helpful if you could revert the changes to the test suite before merging.

Also, I apologize for leaving this in draft status for so long. I would like to introduce git submodule/sparse-checkout to WPT and work on making it easier to update test suite, so please wait a little longer.

Sure thing. I can add a branch with just the Crypto changes excluding the WPT changes

richarddavison avatar May 12 '25 11:05 richarddavison

@richarddavison I will be away from home on a business trip for about three days, so I won't be able to make any progress during that time, but it's working to some extent so I'd be happy if you could give it a try.

  1. Update sparse-checkout information (Only the first one) $ make init-wpt

  2. Loading as a submodule (Only the first one) ~$ git submodule update --init~ $ git submodule add --force -b master https://github.com/web-platform-tests/wpt tests/wpt/wpt

  3. Update the local revision (when the remote is updated) $ make update-wpt

  4. Running WPTs $ make test-wpt

nabetti1720 avatar May 12 '25 13:05 nabetti1720

@richarddavison It's not perfect yet, and we plan to continue improving it. But the tests that currently work are able to detect many incompatibilities, so it's definitely worth merging it. :)

nabetti1720 avatar May 16 '25 10:05 nabetti1720

@richarddavison It's not perfect yet, and we plan to continue improving it. But the tests that currently work are able to detect many incompatibilities, so it's definitely worth merging it. :)

Yes for sure. Thanks for this amazing contribution!

richarddavison avatar May 16 '25 10:05 richarddavison

@nabetti1720 what's the state of this PR? Is it ready for review? I've not looked to close yet but so far it looks great!

richarddavison avatar May 16 '25 21:05 richarddavison

@richarddavison Yes, I've finished what I wanted to do with this PR, so you can review it any time.

nabetti1720 avatar May 16 '25 22:05 nabetti1720

@richarddavison I'm sorry. I was concerned about the directory structure, so I revised it to make the external repository more independent. This is really the end. :)

nabetti1720 avatar May 21 '25 12:05 nabetti1720

Fantastic! Sorry for the delay, been traveling for Summit season :) I'll take a proper look over the weekend!

richarddavison avatar May 23 '25 20:05 richarddavison

@richarddavison Thank you for your continued support. I am currently working on improving llrt_fetch little by little, taking advantage of the results of this PR. :)

nabetti1720 avatar May 24 '25 01:05 nabetti1720

@richarddavison Inspired by the test262 record of quickjs, I decided to leave a wpt_errors.txt file. Until now, it was very difficult to see the regressions caused by my own modifications, but now you can see them as git diffs. This will definitely be the last fix in this PR.

nabetti1720 avatar May 25 '25 07:05 nabetti1720

Sorry for the delay and thanks for the fantastic contribution!!!

richarddavison avatar May 27 '25 06:05 richarddavison