FreeBSD ARM64 CI Failures

[nickva]

Noticed a few consistent CI failures on FreeBSD ARM64 runner:

[2023-12-13T01:06:13.961Z]     chttpd_view_test:89: -view_test_/0-fun-0- (t_view_map_reduce_with_keys_and_group)...[0.010 s] ok
[2023-12-13T01:06:13.961Z] erl_child_setup: failed with error 32 on line 281
[2023-12-13T01:06:13.961Z] erl_child_setup: failed with error 32 on line 281
[2023-12-13T01:06:13.961Z]     [done in 0.934 s]
[2023-12-13T01:06:13.961Z]   chttpd all docs tests
[2023-12-13T01:06:14.493Z]     undefined
[2023-12-13T01:06:14.493Z]     *** context setup failed ***
[2023-12-13T01:06:14.493Z] **in function chttpd_view_test:create_ddoc/1 (test/eunit/chttpd_view_test.erl, line 400)
[2023-12-13T01:06:14.493Z] in call from chttpd_view_test:setup/0 (test/eunit/chttpd_view_test.erl, line 55)
[2023-12-13T01:06:14.493Z] **error:{failed_to_create_ddocs,<<"eunit-test-db-c861572e65bff0e2f5ca0b703bc2c286">>,
[2023-12-13T01:06:14.493Z]                         {400,
[2023-12-13T01:06:14.493Z]                          #{<<"error">> => <<"compilation_error">>,
[2023-12-13T01:06:14.493Z]                            <<"reason">> =>
[2023-12-13T01:06:14.493Z]                                <<"Compilation of the map function in the 'map' view fa"...>>}}}

[2023-12-13T02:46:42.337Z]   Upgrade and bugs related tests
[2023-12-13T02:46:42.337Z]     undefined
[2023-12-13T02:46:42.337Z]     *** context setup failed ***
[2023-12-13T02:46:42.337Z] **in function couch_db:update_doc/4 (src/couch_db.erl, line 875)
[2023-12-13T02:46:42.337Z] in call from couchdb_views_tests:create_design_doc/3 (test/eunit/couchdb_views_tests.erl, line 849)
[2023-12-13T02:46:42.337Z] in call from couchdb_views_tests:setup_with_docs/0 (test/eunit/couchdb_views_tests.erl, line 40)
[2023-12-13T02:46:42.337Z] **throw:{bad_request,compilation_error,
[2023-12-13T02:46:42.337Z]              [67,111,109,112,105,108,97,116,105,111,110,32,111,102,32,116|...]}
[2023-12-13T02:46:42.337Z] 
[2023-12-13T02:46:42.337Z] 
[2023-12-13T02:46:42.337Z]   [done in 0.060 s]

[2023-12-13T03:25:45.444Z]   Test couchjs
[2023-12-13T03:25:45.444Z]     couch_js_tests:97: -with/1-fun-0- (should_create_sandbox)...[0.058 s] ok
[2023-12-13T03:25:45.444Z]     couch_js_tests:97: -with/1-fun-0- (should_reset_properly)...[0.011 s] ok
[2023-12-13T03:25:45.444Z]     couch_js_tests:97: -with/1-fun-0- (should_freeze_doc_object)...[0.003 s] ok
[2023-12-13T03:25:45.444Z]     couch_js_tests:97: -with/1-fun-0- (should_roundtrip_utf8)...[0.003 s] ok
[2023-12-13T03:25:45.444Z]     couch_js_tests:97: -with/1-fun-0- (should_roundtrip_modified_utf8)...[0.002 s] ok
[2023-12-13T03:25:45.444Z]     couch_js_tests:97: -with/1-fun-0- (should_replace_broken_utf16)...[0.002 s] ok
[2023-12-13T03:25:45.444Z]     couch_js_tests:97: -with/1-fun-0- (should_allow_js_string_mutations)...[0.005 s] ok
[2023-12-13T03:25:45.444Z]     couch_js_tests:97: -with/1-fun-0- (should_bump_timing_and_call_stats)...[0.010 s] ok
[2023-12-13T03:25:45.444Z]     couch_js_tests:99: -with/1-fun-1- (should_exit_on_oom)...*failed*
[2023-12-13T03:25:45.444Z] in function couch_os_process:prompt/2 (src/couch_os_process.erl, line 47)
[2023-12-13T03:25:45.444Z] in call from couch_query_servers:proc_prompt/2 (src/couch_query_servers.erl, line 601)
[2023-12-13T03:25:45.444Z] in call from couch_js_tests:trigger_oom/1 (test/eunit/couch_js_tests.erl, line 406)
[2023-12-13T03:25:45.444Z] in call from eunit_test:run_testfun/1 (eunit_test.erl, line 71)
[2023-12-13T03:25:45.444Z] in call from eunit_proc:run_test/1 (eunit_proc.erl, line 531)
[2023-12-13T03:25:45.444Z] in call from eunit_proc:with_timeout/3 (eunit_proc.erl, line 356)
[2023-12-13T03:25:45.444Z] in call from eunit_proc:handle_test/2 (eunit_proc.erl, line 514)
[2023-12-13T03:25:45.444Z] in call from eunit_proc:tests_inorder/3 (eunit_proc.erl, line 456)
[2023-12-13T03:25:45.444Z] **throw:{os_process_error,{exit_status,139}}
[2023-12-13T03:25:45.444Z]   output:<<"">>
[2023-12-13T03:25:45.444Z] 

[nickva]

Another one. This one doesn't explicitly show a couch_js execution error but it failed to build an index:

[2023-12-13T04:34:11.011Z] module 'couch_mrview_ddoc_updated_tests'
[2023-12-13T04:34:11.011Z]   Check ddoc update actions
[2023-12-13T04:34:11.011Z]     undefined
[2023-12-13T04:34:11.011Z]     *** context setup failed ***
[2023-12-13T04:34:11.011Z] **in function couch_mrview_util:get_view_index_state/5 (src/couch_mrview_util.erl, line 219)
[2023-12-13T04:34:11.011Z] in call from couch_mrview_util:get_view/4 (src/couch_mrview_util.erl, line 153)
[2023-12-13T04:34:11.011Z] in call from couch_mrview:query_view/6 (src/couch_mrview.erl, line 286)
[2023-12-13T04:34:11.011Z] in call from couch_mrview_ddoc_updated_tests:setup/0 (test/eunit/couch_mrview_ddoc_updated_tests.erl, line 50)
[2023-12-13T04:34:11.011Z] **throw:{not_found,missing}
[2023-12-13T04:34:11.011Z] 

[nickva]

Another example:

[2023-12-13T19:12:08.828Z] module 'couchdb_design_doc_tests'
[2023-12-13T19:12:08.828Z]   Check _list functionality
[2023-12-13T19:12:09.390Z]     undefined
[2023-12-13T19:12:09.390Z]     *** context setup failed ***
[2023-12-13T19:12:09.390Z] **in function couch_db:update_doc/4 (src/couch_db.erl, line 875)
[2023-12-13T19:12:09.390Z] in call from couchdb_design_doc_tests:create_design_doc/2 (test/eunit/couchdb_design_doc_tests.erl, line 90)
[2023-12-13T19:12:09.390Z] in call from couchdb_design_doc_tests:setup/0 (test/eunit/couchdb_design_doc_tests.erl, line 22)
[2023-12-13T19:12:09.390Z] **throw:{bad_request,compilation_error,
[2023-12-13T19:12:09.390Z]              [67,111,109,112,105,108,97,116,105,111,110,32,111,102,32,116|...]}
[2023-12-13T19:12:09.390Z] 
[2023-12-13T19:12:09.390Z] 
[2023-12-13T19:12:09.390Z]   couchdb_design_doc_tests:60: -should_return_empty_when_no_docs/1-fun-1-...[0.068 s] ok
[2023-12-13T19:12:09.390Z]   [done in 0.360 s]
[2023-12-13T19:12:09.390Z] [done in 0.477 s]

[dch]

these are caused by a SIGSEGV in couchjs, during GC. worker threads are paused, main thread is in the middle of GC. Filed upstream but they rightly point out we need to be on a SM supported version for this to be worth looking at.

* thread #1, name = 'couchjs', stop reason = signal SIGSEGV
  * frame #0: 0x000000008321f3a0 libmozjs-91.so`js::gc::MapAlignedPages(unsigned long, unsigned long) + 1232
    frame #1: 0x00000000831e56f0 libmozjs-91.so`js::gc::GCRuntime::pickChunk(js::AutoLockGCBgAlloc&) + 132
    frame #2: 0x00000000831e54a4 libmozjs-91.so`js::gc::ArenaLists::refillFreeListAndAllocate(js::gc::FreeLists&, js::gc::AllocKind, js::gc::ShouldCheckThresholds) + 308
    frame #3: 0x00000000831e4638 libmozjs-91.so`js::jit::JitCode* js::Allocate<js::jit::JitCode, (js::AllowGC)0>(JSContext*) + 128
    frame #4: 0x0000000083478618 libmozjs-91.so`js::jit::JitCode* js::jit::JitCode::New<(js::AllowGC)0>(JSContext*, unsigned char*, unsigned int, unsigned int, js::jit::ExecutablePool*, js::jit::CodeKind) + 44
    frame #5: 0x00000000834a27c8 libmozjs-91.so`js::jit::Linker::newCode(JSContext*, js::jit::CodeKind) + 284
    frame #6: 0x00000000832922c8 libmozjs-91.so`js::jit::BaselineCacheIRCompiler::compile() + 15424
    frame #7: 0x000000008329b6e0 libmozjs-91.so`js::jit::AttachBaselineCacheIRStub(JSContext*, js::jit::CacheIRWriter const&, js::jit::CacheKind, JSScript*, js::jit::ICScript*, js::jit::ICFallbackStub*, bool*) + 212
    frame #8: 0x00000000832cf2a0 libmozjs-91.so`___lldb_unnamed_symbol30268 + 276
    frame #9: 0x00000000832cf070 libmozjs-91.so`js::jit::DoGetPropFallback(JSContext*, js::jit::BaselineFrame*, js::jit::ICFallbackStub*, JS::MutableHandle<JS::Value>, JS::MutableHandle<JS::Value>) + 444
    frame #10: 0x000014c25c2ea554

I will see if I can track down what dependencies changed back in December, that this stopped working. couchjs is not exactly a fast-moving target.

[nickva]

I believe it works now, we have the ARM64 FreeBSD worker back in full CI rotation.