ponyc
ponyc copied to clipboard
ponylang
Arm64 Apple Darwin CI test failures
This issue is a place to track failures that occur with AMD64 Apple Darwin tests that are unrelated to changes that were being tested. There's something amiss either in CI setup or an intermittent bug (or bugs) on Arm64 Apple Darwin.
The following "ponyc-runner" tests failed with a timeout:
[ FAILED ]
12 test(s), listed below:
[ FAILED ]
annotation-methods
[ FAILED ]
regression-3840
[ FAILED ]
array-infer-from-values-call-with-iterator-antecedent
[ FAILED ]
bare-function-bare-lambda-callback
[ FAILED ]
array-infer-from-array-type
[ FAILED ]
identity-digestof-boxed-tuple
[ FAILED ]
with-in-loop-with-continue
[ FAILED ]
c-callback
[ FAILED ]
iftype-false
[ FAILED ]
class-final
[ FAILED ]
regression-3615
[ FAILED ]
identity-boxed-numeric-is-boxed-numeric
Note they were all run around the same time and then everything else passed again. Perhaps a pause in CPU availability in the CI environment that caused them all the fail.
I've attached the complete log as well. release_test.log.txt
Similar timeout failures clumped together in #4146:
[ RUN ] identity-tuple-is-boxed-tuple
[ FAILED ] identity-tuple-is-boxed-tuple (60011 ms): timed out after 60 seconds
[ RUN ] array-infer-from-apply-interface-box-element-in-tuple
[ FAILED ] array-infer-from-apply-interface-box-element-in-tuple (60001 ms): timed out after 60 seconds
[ RUN ] actor-final
[ FAILED ] actor-final (60002 ms): timed out after 60 seconds
[ RUN ] iftype-else-if-true
[ FAILED ] iftype-else-if-true (60012 ms): timed out after 60 seconds
[ RUN ] iftype-true
[ FAILED ] iftype-true (60034 ms): timed out after 60 seconds
[ RUN ] while-else-trigger-else-clause
[ FAILED ] while-else-trigger-else-clause (60005 ms): timed out after 60 seconds
[ RUN ] small-finalisers
[ FAILED ] small-finalisers (60010 ms): timed out after 60 seconds
[ RUN ] codegen-trace-struct-field
[ FAILED ] codegen-trace-struct-field (60006 ms): timed out after 60 seconds
[ RUN ] identity-abstract-type-no-subtyping
[ FAILED ] identity-abstract-type-no-subtyping (60010 ms): timed out after 60 seconds
[ RUN ] lambda-with-return-in-behaviour
[ FAILED ] lambda-with-return-in-behaviour (60002 ms): timed out after 60 seconds
[ RUN ] identity-digestof-object
[ FAILED ] identity-digestof-object (60014 ms): timed out after 60 seconds
Failures from PR #4150
[ FAILED ]
12 test(s), listed below:
[ FAILED ] regression-2408
[ FAILED ] match-exhaustive-all-cases-including-dont-care-and-tuple
[ FAILED ] custom-serialisation
[ FAILED ] array-infer-from-values-of-array-interface
[ FAILED ] regression-849
[ FAILED ] try-then-clause-break
[ FAILED ] codegen-trace-number-boxed
[ FAILED ] array-infer-most-specific-from-union-of-array-types
[ FAILED ] primitive-init
[ FAILED ] array-infer-from-seq
[ FAILED ] codegen-trace-tuple-boxed-sent-through-interface
[ FAILED ] try-else-can-initialize-fields
make: *** [test-full-programs-debug] Error 1
Exit status: 2
Failures from #4149
[ RUN ] codegen-trace-number-boxed-sent-through-interface
[ FAILED ] codegen-trace-number-boxed-sent-through-interface (60002 ms): timed out after 60 seconds
[ RUN ] ffi-return-arg-reachable
[ FAILED ] ffi-return-arg-reachable (60011 ms): timed out after 60 seconds
[ RUN ] bare-function-call
[ FAILED ] bare-function-call (60037 ms): timed out after 60 seconds
[ RUN ] regression-3447
[ FAILED ] regression-3447 (60010 ms): timed out after 60 seconds
[ RUN ] codegen-trace-tuple-boxed
[ FAILED ] codegen-trace-tuple-boxed (59984 ms): timed out after 60 seconds
[ RUN ] try-then-clause-continue
[ FAILED ] try-then-clause-continue (60018 ms): timed out after 60 seconds
[ RUN ] array-infer-from-array-val
[ FAILED ] array-infer-from-array-val (60008 ms): timed out after 60 seconds
[ RUN ] codegen-trace-object-dynamic
[ FAILED ] codegen-trace-object-dynamic (60001 ms): timed out after 60 seconds
[ RUN ] if-block-ending-with-dont-care-assign
[ FAILED ] if-block-ending-with-dont-care-assign (60029 ms): timed out after 60 seconds
[ RUN ] array-no-as-iso-consumed-elements
[ FAILED ] array-no-as-iso-consumed-elements (60013 ms): timed out after 60 seconds
[ RUN ] identity-tuple-cardinality
[ FAILED ] identity-tuple-cardinality (60023 ms): timed out after 60 seconds
[ RUN ] object-literal-with-return-in-behaviour
[ FAILED ] object-literal-with-return-in-behaviour (60005 ms): timed out after 60 seconds
From #4150
[ RUN ] array-infer-from-seq
[ FAILED ] array-infer-from-seq (60001 ms): timed out after 60 seconds
[ RUN ] regression-849
[ FAILED ] regression-849 (60005 ms): timed out after 60 seconds
[ RUN ] try-else-can-initialize-fields
[ FAILED ] try-else-can-initialize-fields (60008 ms): timed out after 60 seconds
[ RUN ] try-then-clause-break
[ FAILED ] try-then-clause-break (60021 ms): timed out after 60 seconds
[ RUN ] codegen-trace-number-boxed
[ FAILED ] codegen-trace-number-boxed (60001 ms): timed out after 60 seconds
[ RUN ] custom-serialisation
[ FAILED ] custom-serialisation (60010 ms): timed out after 60 seconds
[ RUN ] array-infer-most-specific-from-union-of-array-types
[ FAILED ] array-infer-most-specific-from-union-of-array-types (60009 ms): timed out after 60 seconds
[ RUN ] regression-2408
[ FAILED ] regression-2408 (60004 ms): timed out after 60 seconds
[ RUN ] codegen-trace-tuple-boxed-sent-through-interface
[ FAILED ] codegen-trace-tuple-boxed-sent-through-interface (60000 ms): timed out after 60 seconds
[ RUN ] array-infer-from-values-of-array-interface
[ FAILED ] array-infer-from-values-of-array-interface (59999 ms): timed out after 60 seconds
[ RUN ] regression-1222
[ OK ] regression-1222 (58494 ms)
[ RUN ] identity-digestof-tuple
[ OK ] identity-digestof-tuple (57664 ms)
[ RUN ] union-value-for-tuple-return-type
[ OK ] union-value-for-tuple-return-type (57776 ms)
[ RUN ] match-exhaustive-all-cases-including-dont-care-and-tuple
[ FAILED ] match-exhaustive-all-cases-including-dont-care-and-tuple (60010 ms): timed out after 60 seconds
[ RUN ] codegen-trace-no-trace
[ OK ] codegen-trace-no-trace (58108 ms)
[ RUN ] ffi-different-returns
[ FAILED ] ffi-different-returns (59964 ms): timed out after 60 second
Aside from https://github.com/ponylang/ponyc/issues/4138#issuecomment-1167699736, I am convinced that this is environmental; a vm losing CPU access. However https://github.com/ponylang/ponyc/issues/4138#issuecomment-1167699736 gives me pause and wondering if that is entirely true. I'm still believing that this is environmentally though.
At sync we discussed a bit more and decided that https://github.com/ponylang/ponyc/issues/4138#issuecomment-1167699736 is probably also environmental where a pause of almost 60 seconds would explain the failures.
There's no much we can do for this but track for now and let Cirrus know what we are seeing.
What do you mean CPU access? Like a VM freezes? Then all of the process should've been paused I think.
Also do you mind providing links to Cirrus tasks. I have hard time figuring them out from the PR links. Seems some other failures were on Arm Linux instances as well.
@fkorotkov it would appear that some tests are started and then all of them that are running time out. The only thing looking at when and how this happens is that everything executing stops and when starting again, all of or most of the timeout has passed. Given the weird little batches it appears in, it feels like the vm stops running. I don't know the particulars of the architecture but this feels like when vms are overprovisioned and every now and then don't have access to the CPU. I doubt that is the architecture here but it definitely looks like it.
I don't have specific logs but could go digging some up based on looking at the PRs linked above, finding the branch name for the PR and combing through the different runs looking for the specific failures.
@fkorotkov these are all Apple silicon failures. I haven't seen anything like this in any other test environments and yes, this looks like the entire vm froze and when it came back, the timeout had passed and the test runner that started again timed out all of the tests that were running when the freeze occurred.
I suspect if we turned off timeouts for Apple arm that these failures would go away as the tests would pass. However that would be problematic for the scenario where a problem was introduced.
All in all, this is a minor annoyance to us and can be easily distinguished from "real failures" so I'm not concerned for us, but in thought you'd be interested in the report given it's a new platform and tech for you.
@ponylang/committer the next time you encounter this issue, please don't restart the job. leave it as is as it appears a "rerun job" make impossible for me to find the failure in CirrusCI after the fact.
@fkorotkov it would appear that via the UI, once we do "rerun failure" that there's no way in the UI to get the given run that failed.