function_clause in chttpd_db:update_doc/4

[wohali]

Description

Single node CouchDB under light load experiences this error occasionally:

[error] 2019-09-20T17:42:46.575000Z couchdb@localhost <0.2596.287> -------- rpc couch_db:update_docs/4 function_clause [{couch_att,flush_data,[{db,1,<<"shards/c0000000-dfffffff/reports.1544621486">>,"./data/shards/c0000000-dfffffff/reports.1544621486.couch",{couch_bt_engine,{st,"./data/shards/c0000000-dfffffff/reports.1544621486.couch",<0.1004.0>,#Ref<0.0.24117249.50842>,[before_header,after_header,on_file_open],{db_header,7,4854,0,{1051727219,{3,44,{size_info,322320088,255257436}},348445},{1052064637,47,348371},{382404,[],9691},nil,nil,303855,1000,<<"0971df3c9d732b38eca6ad92cde44404">>,[{couchdb@localhost,0}],3359,1000},false,{btree,<0.1004.0>,{1051727219,{3,44,{size_info,322320088,255257436}},348445},#Fun<couch_bt_engine.id_tree_split.1>,#Fun<couch_bt_engine.id_tree_join.2>,undefined,#Fun<couch_bt_engine.id_tree_reduce.2>,snappy},{btree,<0.1004.0>,{1052064637,47,348371},#Fun<couch_bt_engine.seq_tree_split.1>,#Fun<couch_bt_engine.seq_tree_join.2>,undefined,#Fun<couch_bt_engine.seq_tree_reduce.2>,snappy},{btree,<0.1004.0>,{382404,[],9691},#Fun<couch_bt_engine.local_tree_split.1>,#Fun<couch_bt_engine.local_tree_join.2>,undefined,nil,snappy},snappy,{btree,<0.1004.0>,nil,#Fun<couch_bt_engine.purge_tree_split.1>,#Fun<couch_bt_engine.purge_tree_join.2>,undefined,#Fun<couch_bt_engine.purge_tree_reduce.2>,snappy},{btree,<0.1004.0>,nil,#Fun<couch_bt_engine.purge_seq_tree_split.1>,#Fun<couch_bt_engine.purge_seq_tree_join.2>,undefined,#Fun<couch_bt_engine.purge_tree_reduce.2>,snappy}}},<0.1003.0>,nil,4854,<<"1568841471431000">>,{user_ctx,<<"admin">>,[<<"_admin">>],<<"default">>},[{<<"admins">>,{[{<<"names">>,[<<"mozaik_system">>,<<"admin">>]}]}}],[],nil,nil,nil,[{default_security_object,[]},{create_if_missing,true}],undefined},stub,{att,<<"bfp-1|B0.2">>,<<"image/jpeg">>,737414,737414,<<27,251,102,250,238,111,104,77,141,31,77,235,33,208,150,78>>,1,stub,identity}],[{file,"src/couch_att.erl"},{line,534}]},{couch_db,'-doc_flush_atts/2-lc$^0/1-0-',2,[{file,"src/couch_db.erl"},{line,1365}]},{couch_db,doc_flush_atts,2,[{file,"src/couch_db.erl"},{line,1365}]},{couch_db,'-update_docs/4-lc$^4/1-4-',2,[{file,"src/couch_db.erl"},{line,1184}]},{couch_db,'-update_docs/4-lc$^3/1-3-',2,[{file,"src/couch_db.erl"},{line,1183}]},{couch_db,update_docs,4,[{file,"src/couch_db.erl"},{line,1183}]},{fabric_rpc,with_db,3,[{file,"src/fabric_rpc.erl"},{line,334}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,140}]}]
[error] 2019-09-20T17:42:46.575000Z couchdb@localhost <0.889.287> c6691696a0 req_err(181230657) unknown_error : function_clause
    [<<"chttpd_db:update_doc/4 L1196">>,<<"chttpd_db:send_updated_doc/6 L1160">>,<<"chttpd:handle_req_after_auth/2 L320">>,<<"chttpd:process_request/1 L303">>,<<"chttpd:handle_request_int/1 L243">>,<<"mochiweb_http:headers/6 L128">>,<<"proc_lib:init_p_do_apply/3 L247">>]

Steps to Reproduce

Unknown.

Expected Behaviour

function_clause is always a bug.

Your Environment

• CouchDB Version used: 2.3.1
• Browser name and version: n/a
• Operating System and version: Windows Server 2008

[celder628]

We are seeing an issue on CouchDB 2.3.1 and were curious if this is the same or a related problem:

There are a number of GETs that seems to run quickly. One of the calls is a query:

2022-08-25T09:32:42.672765857Z [notice] 2022-08-25T09:32:42.672090Z nonode@nohost <0.1173.0> 136e1a264a localhost:5984 127.0.0.1 J3AnRJkA8PjiQ1nedk0s2Bi8FiUSEsg5 POST /jci-prod-channel_jcicc/_all_docs?include_docs=true 200 ok 4

About 10 seconds later we see:

2022-08-25T09:32:53.374978586Z [error] 2022-08-25T09:32:53.374140Z nonode@nohost <0.8108.2> eb8d1a741a req_err(1041696419) unknown_error : function_clause

2022-08-25T09:32:53.375007768Z [<<"couch_mrview_http:view_cb/2 L338">>,<<"chttpd_db:all_docs_view/4 L769">>,<<"chttpd:handle_req_after_auth/2 L320">>,<<"chttpd:process_request/1 L303">>,<<"chttpd:handle_request_int/1 L243">>,<<"mochiweb_http:headers/6 L128">>,<<"proc_lib:init_p_do_apply/3 L247">>]

2022-08-25T09:32:53.375023894Z [notice] 2022-08-25T09:32:53.374479Z nonode@nohost <0.8108.2> eb8d1a741a localhost:5984 127.0.0.1 J3AnRJkA8PjiQ1nedk0s2Bi8FiUSEsg5 POST /jci-prod-channel_lscc/_all_docs?include_docs=true 500 ok 10707

The couchdb server does not appear to be under any stress when this happens. Since this is from a customer, we don't have the exact query that is being executed. We're trying to evaluate using logs.

Is it possible this is a bad query causing the error in the background?

[denyeart]

@wohali I'm working with @celder628 on this one. Any advice on how to troubleshoot these random function_clause errors on relatively idle CouchDB 2.3.1 servers? Would you expect improvement on CouchDB v3.2?

[nickva]

You could try adding an extra function clause to couch_mrview_http:view_cb/2 to log the error with couch_log:error(...).

view_cb({row, Row}, #vacc{meta_sent=true}=Acc) ->
    % Adding another row
    Chunk = [prepend_val(Acc), row_to_json(Row)],
    maybe_flush_response(Acc#vacc{row_sent=true}, Chunk, iolist_size(Chunk));

view_cb(Msg, Acc) ->
    couch_log:error("~p : xxxxx function_clause ~p ~p", [?MODULE, Msg, Acc]),
    error(function_clause).

There is also a good chance it was fixed in 3.x series, so try to upgrade to 3.2.2 and see if it still happens.

[big-r81]

Closing this (CouchDB 2.3.1).