ruffle icon indicating copy to clipboard operation
ruffle copied to clipboard

ci: Set the "render" kind on tests that need renderer, mark them as heavy for nextest

Open torokati44 opened this issue 1 year ago • 13 comments

~~But it doesn't seem to work locally for me...? :face_with_spiral_eyes:~~

Turns out, updating cargo-nextest fixed it for me...

Relevant docs pages: https://nexte.st/docs/configuration/threads-required/ https://nexte.st/docs/filtersets/reference/

This will hopefully stop random crashes of visual tests (especially on avm2/pixelbender_shaderdata and avm2/graphics_round_rects) on GHA.

torokati44 avatar Aug 19 '24 18:08 torokati44

I think I'd prefer a suffix, usually things are module::path::testname, so this kinda fits

Dinnerbone avatar Aug 19 '24 19:08 Dinnerbone

Yeah, that's what I tried first, but... image

It kinda makes render look like it's the test name, and the test name like it's something else...?

torokati44 avatar Aug 19 '24 19:08 torokati44

Hmmm okay. Prefix looks fine then

Dinnerbone avatar Aug 19 '24 19:08 Dinnerbone

Cool! Does this count as a green checkmark? 🤔

torokati44 avatar Aug 19 '24 20:08 torokati44

Good catch, thanks!

Can a test have any number of kinds though?

torokati44 avatar Aug 20 '24 15:08 torokati44

Can a test have any number of kinds though?

A test may have one kind only, but we can join multiple kinds with a comma (I think, not entirely sure it'll work) and add support for multiple kinds that way ourselves.

kjarosh avatar Aug 20 '24 15:08 kjarosh

Now thinking about it... a test kind is just a string prefix (but inside brackets), which is ignored when looking up tests. I think we can safely put multiple strings separated by a comma there and it'll work, even if we pass no kind or an invalid kind when looking up tests.

The only situation when the contents of a kind matter is when we parse them, so in .config/nextest.toml and possibly in the code.

kjarosh avatar Aug 20 '24 15:08 kjarosh

Okay, stole some bits and pieces from https://github.com/ruffle-rs/ruffle/pull/15950, now render is a kind, not a name prefix.

torokati44 avatar Aug 20 '24 15:08 torokati44

Aaaand now it automatically does the [render] tests in one group first, so it makes the failure more likely... Let's try a weight of 3, so no two render tests can run at the same time, but other ones can together with them...?

torokati44 avatar Aug 20 '24 16:08 torokati44

Even one at a time they crash... 😨😵😵‍💫🫥🫠

torokati44 avatar Aug 20 '24 16:08 torokati44

Maybe just set a higher number of retries for render tests for now?

kjarosh avatar Aug 20 '24 17:08 kjarosh

Maybe just set a higher number of retries for render tests for now?

Interestingly, it's again only the same two tests that crashed. So I'd allow retries only for those then - but then we're almost back to square one.

torokati44 avatar Aug 20 '24 17:08 torokati44

Finally, a stack trace!

https://github.com/ruffle-rs/ruffle/actions/runs/10488227410/job/29050242404?pr=17574#step:8:7584

Thread 11 (Thread 0x7fffc7e006c0 (LWP 14650) "tests-6cab02727"):
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3  0x00007ffff7c4526e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x00007ffff7c288ff in __GI_abort () at ./stdlib/abort.c:79
#5  0x00007ffff7c297b6 in __libc_message_impl (fmt=fmt@entry=0x7ffff7dce8d7 "%s\n") at ../sysdeps/posix/libc_fatal.c:132
#6  0x00007ffff7ca8fe5 in malloc_printerr (str=str@entry=0x7ffff7dd1ae0 "double free or corruption (!prev)") at ./malloc/malloc.c:5772
#7  0x00007ffff7cab11c in _int_free_merge_chunk (av=0x7ffff7e03ac0 <main_arena>, p=0x5555575a0020, size=1040) at ./malloc/malloc.c:4679
#8  0x00007ffff7cab42a in _int_free (av=0x7ffff7e03ac0 <main_arena>, p=<optimized out>, have_lock=<optimized out>) at ./malloc/malloc.c:4646
#9  0x00007ffff7cadd9e in __GI___libc_free (mem=0x5555575a0030) at ./malloc/malloc.c:3398
#10 0x00007fffeee63f17 in void std::vector<std::unique_ptr<llvm::PassInfo const, std::default_delete<llvm::PassInfo const> >, std::allocator<std::unique_ptr<llvm::PassInfo const, std::default_delete<llvm::PassInfo const> > > >::_M_realloc_insert<std::unique_ptr<llvm::PassInfo const, std::default_delete<llvm::PassInfo const> > >(__gnu_cxx::__normal_iterator<std::unique_ptr<llvm::PassInfo const, std::default_delete<llvm::PassInfo const> >*, std::vector<std::unique_ptr<llvm::PassInfo const, std::default_delete<llvm::PassInfo const> >, std::allocator<std::unique_ptr<llvm::PassInfo const, std::default_delete<llvm::PassInfo const> > > > >, std::unique_ptr<llvm::PassInfo const, std::default_delete<llvm::PassInfo const> >&&) () from /lib/x86_64-linux-gnu/libLLVM-17.so.1
#11 0x00007fffeee63443 in llvm::PassRegistry::registerPass(llvm::PassInfo const&, bool) () from /lib/x86_64-linux-gnu/libLLVM-17.so.1
#12 0x00007fffeef94ec7 in ?? () from /lib/x86_64-linux-gnu/libLLVM-17.so.1
#13 0x00007ffff7ca1ec3 in __pthread_once_slow (once_control=0x7ffff55fc194, init_routine=0x7ffff64e93b0 <__once_proxy>) at ./nptl/pthread_once.c:116
#14 0x00007fffeef94ded in llvm::initializeEarlyIfConverterPass(llvm::PassRegistry&) () from /lib/x86_64-linux-gnu/libLLVM-17.so.1
#15 0x00007fffeef40042 in llvm::initializeCodeGen(llvm::PassRegistry&) () from /lib/x86_64-linux-gnu/libLLVM-17.so.1
#16 0x00007fffef2c3dec in llvm::TargetPassConfig::TargetPassConfig(llvm::LLVMTargetMachine&, llvm::legacy::PassManagerBase&) () from /lib/x86_64-linux-gnu/libLLVM-17.so.1
#17 0x00007ffff1a1727f in ?? () from /lib/x86_64-linux-gnu/libLLVM-17.so.1
#18 0x00007fffef030d5a in llvm::LLVMTargetMachine::addPassesToEmitMC(llvm::legacy::PassManagerBase&, llvm::MCContext*&, llvm::raw_pwrite_stream&, bool) () from /lib/x86_64-linux-gnu/libLLVM-17.so.1
#19 0x00007ffff0837ac9 in llvm::MCJIT::emitObject(llvm::Module*) () from /lib/x86_64-linux-gnu/libLLVM-17.so.1
#20 0x00007ffff0837c6d in llvm::MCJIT::generateCodeForModule(llvm::Module*) () from /lib/x86_64-linux-gnu/libLLVM-17.so.1
#21 0x00007ffff08386ed in llvm::MCJIT::finalizeObject() () from /lib/x86_64-linux-gnu/libLLVM-17.so.1
#22 0x00007ffff07bf287 in LLVMGetPointerToGlobal () from /lib/x86_64-linux-gnu/libLLVM-17.so.1

"double free or corruption (!prev)")

Hmm, LLVM bug?

torokati44 avatar Aug 21 '24 10:08 torokati44

Cross-referencing, about the crash: https://gitlab.freedesktop.org/mesa/mesa/-/issues/11797

torokati44 avatar Aug 29 '24 01:08 torokati44