serenity icon indicating copy to clipboard operation
serenity copied to clipboard

Published 20 hours ago • verified publisher icon SerenityOS

LibJS: test262 cases that fail with ASAN and UBSAN enabled

Open ADKaster opened this issue 3 years ago • 3 comments

Here's a list of test cases that fail when libjs-test262 test cases are run with ASAN.

These failing tests likely represent some gaps in the LibJS test suite in the main serenity repository.

The following steps will reproduce how I ran the tests compared to a pre-asan run with the default run options from the README:

./setup.sh
cmake -B Build -DENABLE_ADDRESS_SANITIZER=ON -DENABLE_UNDEFINED_SANITIZER=ON
ninja -C Build
export ASAN_OPTIONS=strict_string_checks=1:check_initialization_order=1:strict_init_order=1:allocator_may_return_null=1
export UBSAN_OPTIONS=print_stacktrace=1:print_summary=1:halt_on_error=1
python3 main.py --libjs-test262-runner ./Build/libjs-test262-runner --test262-root ./test262/ --json --per-file -c 16 -s --memory-limit -1 --timeout 60 > asan.json
 python3  ./per_file_result_diff.py -o pre-asan.json -n asan.json

Summary:
   Diff Tests:
        -3 ✅    -33 ❌   +2 💀   +34 💥️
  • [x] test/built-ins/ArrayBuffer/allocation-limit.js (Fixed by adding allocator_may_return_null=1 to ASAN_OPTIONS)
  • [ ] test/built-ins/decodeURI/S15.1.3.1_A2.5_T1.js
  • [ ] test/built-ins/decodeURIComponent/S15.1.3.2_A2.5_T1.js

Tail Call Optimization tests:

These cause a stack overflow, because the stack overflow detection check seems to fail with ASAN enabled

  • [ ] test/language/expressions/call/tco-member-args.js
  • [ ] test/language/expressions/call/tco-non-eval-function-dynamic.js
  • [ ] test/language/expressions/call/tco-non-eval-function.js
  • [ ] test/language/expressions/call/tco-non-eval-global.js
  • [ ] test/language/expressions/call/tco-non-eval-with.js
  • [ ] test/language/expressions/coalesce/tco-pos-null.js
  • [ ] test/language/expressions/coalesce/tco-pos-undefined.js
  • [ ] test/language/expressions/comma/tco-final.js
  • [ ] test/language/expressions/conditional/tco-cond.js
  • [ ] test/language/expressions/conditional/tco-pos.js
  • [ ] test/language/expressions/logical-and/tco-right.js
  • [ ] test/language/expressions/logical-or/tco-right.js
  • [ ] test/language/expressions/tagged-template/tco-call.js
  • [ ] test/language/expressions/tagged-template/tco-member.js
  • [ ] test/language/expressions/tco-pos.js
  • [ ] test/language/statements/block/tco-stmt-list.js
  • [ ] test/language/statements/block/tco-stmt.js
  • [ ] test/language/statements/do-while/tco-body.js
  • [ ] test/language/statements/for/tco-const-body.js
  • [ ] test/language/statements/for/tco-let-body.js
  • [ ] test/language/statements/for/tco-lhs-body.js
  • [ ] test/language/statements/for/tco-var-body.js
  • [ ] test/language/statements/if/tco-else-body.js
  • [ ] test/language/statements/if/tco-if-body.js
  • [ ] test/language/statements/labeled/tco.js
  • [ ] test/language/statements/return/tco.js
  • [ ] test/language/statements/switch/tco-case-body-dflt.js
  • [ ] test/language/statements/switch/tco-case-body.js
  • [ ] test/language/statements/switch/tco-dftl-body.js
  • [ ] test/language/statements/try/tco-catch-finally.js
  • [ ] test/language/statements/try/tco-catch.js
  • [ ] test/language/statements/try/tco-finally.js
  • [ ] test/language/statements/while/tco-body.js
Old results from 2021-11-21 
Summary:
   Diff Tests:
        -20 ✅    -78 ❌   +0 ⚠️   +0 📄   +0 ⚙️   +2 💀   +96 💥️   +0 🐍   +0 📝
  • [ ] test/built-ins/SetIteratorPrototype/next/iteration-mutable.js (#11004)
  • [ ] test/language/expressions/await/await-awaits-thenable-not-callable.js (Corrupted execution context stack)
  • [ ] test/language/expressions/await/await-awaits-thenables-that-throw.js
  • [ ] test/built-ins/Temporal/Instant/prototype/equals/instant-string.js
  • [ ] test/built-ins/Temporal/PlainDateTime/prototype/round/smallestunit-plurals-accepted.js
  • [ ] test/language/expressions/await/await-non-promise-thenable.js
  • [ ] test/built-ins/Temporal/ZonedDateTime/prototype/withTimeZone/subclassing-ignored.js
  • [ ] test/language/expressions/await/await-monkey-patched-promise.js
  • [ ] test/built-ins/Temporal/Duration/compare/relativeto-sub-minute-offset.js
  • [ ] test/built-ins/Temporal/Instant/prototype/toZonedDateTime/timezone-string-datetime.js
  • [ ] test/built-ins/Temporal/ZonedDateTime/prototype/withTimeZone/timezone-string-datetime.js
  • [ ] test/built-ins/Temporal/Duration/compare/relativeto-string-zoneddatetime-wrong-offset.js
  • [ ] test/built-ins/Temporal/Instant/prototype/toJSON/basic.js
  • [ ] test/built-ins/Temporal/ZonedDateTime/timezone-string-datetime.js
  • [ ] test/language/expressions/await/await-throws-rejections.js
  • [ ] test/built-ins/Temporal/Now/plainDateISO/timezone-string-datetime.js
  • [ ] test/built-ins/Temporal/Instant/prototype/toZonedDateTimeISO/timezone-string-datetime.js
  • [ ] test/built-ins/Temporal/PlainDateTime/prototype/round/smallestunit-string-shorthand.js
  • [ ] test/built-ins/Temporal/Instant/from/instant-string.js
  • [ ] test/language/expressions/async-function/try-reject-finally-reject.js
  • [ ] test/built-ins/Temporal/Duration/prototype/total/relativeto-string-zoneddatetime-wrong-offset.js
  • [ ] test/built-ins/Temporal/Instant/prototype/until/instant-string.js
  • [ ] test/built-ins/Temporal/TimeZone/prototype/getOffsetNanosecondsFor/instant-string.js
  • [ ] test/built-ins/Temporal/ZonedDateTime/from/timezone-string-datetime.js
  • [ ] test/built-ins/Temporal/Duration/prototype/total/relativeto-sub-minute-offset.js
  • [ ] test/language/expressions/await/syntax-await-has-UnaryExpression.js
  • [ ] test/language/expressions/await/syntax-await-has-UnaryExpression-with-MultiplicativeExpression.js
  • [ ] test/built-ins/Temporal/Duration/prototype/total/timezone-string-datetime.js
  • [ ] test/built-ins/Temporal/Duration/compare/timezone-string-datetime.js
  • [ ] test/built-ins/Temporal/Now/zonedDateTime/timezone-string-datetime.js
  • [ ] test/built-ins/MapIteratorPrototype/next/iteration-mutable.js
  • [ ] test/built-ins/Temporal/TimeZone/prototype/getOffsetStringFor/instant-string.js
  • [ ] test/built-ins/Temporal/PlainDateTime/prototype/toZonedDateTime/timezone-string-datetime.js
  • [ ] test/language/expressions/object/method-definition/async-super-call-param.js
  • [ ] test/built-ins/Temporal/Now/plainDateTime/timezone-string-datetime.js
  • [ ] test/language/expressions/object/method-definition/async-super-call-body.js
  • [ ] test/built-ins/Temporal/TimeZone/from/argument-valid.js
  • [ ] test/language/statements/for-of/map-contract-expand.js
  • [ ] test/built-ins/Temporal/Instant/prototype/since/instant-string.js
  • [ ] test/intl402/Temporal/TimeZone/from/argument-invalid.js
  • [ ] test/built-ins/Set/prototype/forEach/iterates-values-added-after-foreach-begins.js
  • [ ] test/built-ins/Temporal/Now/zonedDateTimeISO/timezone-string-datetime.js
  • [ ] test/built-ins/Temporal/ZonedDateTime/prototype/round/smallestunit-plurals-accepted.js
  • [ ] test/built-ins/Set/prototype/values/values-iteration-mutable.js
  • [ ] test/built-ins/Temporal/TimeZone/prototype/getPlainDateTimeFor/instant-string.js
  • [ ] test/built-ins/Temporal/ZonedDateTime/prototype/equals/timezone-string-datetime.js
  • [ ] test/built-ins/Map/prototype/clear/map-data-list-is-preserved.js
  • [ ] test/built-ins/Temporal/PlainDate/prototype/toZonedDateTime/timezone-string-datetime.js
  • [ ] test/language/expressions/new.target/unary-expr.js
  • [ ] test/built-ins/Temporal/ZonedDateTime/prototype/round/smallestunit-string-shorthand.js
  • [ ] test/built-ins/ArrayBuffer/allocation-limit.js
  • [ ] test/built-ins/Temporal/TimeZone/from/timezone-string-datetime.js
  • [ ] test/language/expressions/async-arrow-function/try-reject-finally-reject.js
  • [ ] test/language/statements/for-of/set-contract-expand.js
  • [ ] test/built-ins/Temporal/ZonedDateTime/compare/timezone-string-datetime.js
  • [ ] test/built-ins/Temporal/Now/plainDateTimeISO/timezone-string-datetime.js
  • [ ] test/built-ins/Temporal/Instant/prototype/toString/timezone-string-datetime.js
  • [ ] test/built-ins/Temporal/Now/plainDate/timezone-string-datetime.js
  • [ ] test/language/expressions/async-arrow-function/arrow-returns-promise.js
  • [ ] test/language/module-code/top-level-await/syntax/for-await-await-expr-array-literal.js
  • [ ] test/built-ins/RegExp/S15.10.2.8_A3_T17.js
  • [ ] test/built-ins/Temporal/Now/plainTimeISO/timezone-string-datetime.js
  • [ ] test/language/statements/class/definition/methods-async-super-call-body.js
  • [ ] test/built-ins/Temporal/Instant/compare/instant-string.js
  • [ ] test/built-ins/Temporal/PlainTime/prototype/toZonedDateTime/timezone-string-datetime.js
  • [ ] test/language/expressions/await/async-await-interleaved.js
  • [ ] test/language/statements/async-function/try-reject-finally-reject.js

Tail Call Optimization tests:

These cause a stack overflow, because the stack overflow detection check seems to fail with ASAN enabled

  • [ ] test/language/statements/switch/tco-case-body-dflt.js
  • [ ] test/language/statements/block/tco-stmt.js
  • [ ] test/language/expressions/comma/tco-final.js
  • [ ] test/language/statements/switch/tco-case-body.js
  • [ ] test/language/statements/block/tco-stmt-list.js
  • [ ] test/language/statements/switch/tco-dftl-body.js
  • [ ] test/language/expressions/coalesce/tco-pos-undefined.js
  • [ ] test/language/expressions/call/tco-member-args.js
  • [ ] test/language/expressions/call/tco-call-args.js
  • [ ] test/language/expressions/call/tco-non-eval-function-dynamic.js
  • [ ] test/language/statements/if/tco-else-body.js
  • [ ] test/language/statements/try/tco-finally.js
  • [ ] test/language/expressions/call/tco-non-eval-function.js
  • [ ] test/language/expressions/call/tco-non-eval-global.js
  • [ ] test/language/expressions/logical-or/tco-right.js
  • [ ] test/language/statements/try/tco-catch-finally.js
  • [ ] test/language/expressions/call/tco-non-eval-with.js
  • [ ] test/language/statements/return/tco.js
  • [ ] test/language/statements/try/tco-catch.js
  • [ ] test/language/statements/if/tco-if-body.js
  • [ ] test/language/statements/for/tco-var-body.js
  • [ ] test/language/statements/for/tco-lhs-body.js
  • [ ] test/language/statements/labeled/tco.js
  • [ ] test/language/statements/for/tco-const-body.js
  • [ ] test/language/statements/do-while/tco-body.js
  • [ ] test/language/statements/for/tco-let-body.js
  • [ ] test/language/statements/while/tco-body.js
  • [ ] test/language/expressions/conditional/tco-cond.js
  • [ ] test/language/expressions/logical-and/tco-right.js
  • [ ] test/language/expressions/conditional/tco-pos.js
  • [ ] test/language/expressions/tagged-template/tco-member.js
  • [ ] test/language/expressions/tco-pos.js
  • [ ] test/language/expressions/coalesce/tco-pos-null.js

ADKaster avatar Nov 22 '21 01:11 ADKaster

All of the tco tests test Tail Call Optimization which we dont do (yet). And these normally fail with the check in VM with the check at https://github.com/SerenityOS/serenity/blob/master/Userland/Libraries/LibJS/Runtime/VM.h#L92. However from some previous testing I remember this is because with ASAN this check no longer works as it thinks the stack is never full thus it just continues. So those are probably "false positives" which we can fix if we can somehow tell the VM not to go into infinite recursion with ASAN.

davidot avatar Nov 22 '21 11:11 davidot

Ah that makes sense. Well, it doesn't make sense why we don't detect the stack overflow in those cases. But, I've moved all those tests to their own section and re-organized the issue from a table into a task list.

ADKaster avatar Nov 23 '21 06:11 ADKaster

I've updated this issue with new results from 2022-06-04

ADKaster avatar Jun 04 '22 21:06 ADKaster