honeydew icon indicating copy to clipboard operation
honeydew copied to clipboard

no match of right hand side value: []

Open Schultzer opened this issue 3 years ago • 2 comments

Hi @koudelka, we are using HoneyDew in production, for our data ingestion. We are using the global Mnesia queue in a multi node setup running nightly jobs and we have notice this MatchError error appearing from time to time:

no match of right hand side value: []

lib/honeydew/queue/mnesia.ex:145 anonymous fn/2 in Honeydew.Queue.Mnesia.ack/2
mnesia_tm.erl:746 :mnesia_tm.non_transaction/5
lib/honeydew/queue/mnesia.ex:144 Honeydew.Queue.Mnesia.ack/2
lib/honeydew/queue.ex:235 Honeydew.Queue.do_ack/2
lib/honeydew/queue.ex:267 Honeydew.Queue.handle_cast/2
gen_server.erl:680 :gen_server.try_dispatch/4
gen_server.erl:756 :gen_server.handle_msg/6
proc_lib.erl:226 :proc_lib.init_p_do_apply/3
GenServer #PID<0.7182.0> terminating ** (stop) an exception was raised: ** (MatchError) no match of right hand side value: [] (honeydew 1.4.6) lib/honeydew/queue/mnesia.ex:145: anonymous fn/2 in Honeydew.Queue.Mnesia.ack/2 (mnesia 4.18) mnesia_tm.erl:746: :mnesia_tm.non_transaction/5 (honeydew 1.4.6) lib/honeydew/queue/mnesia.ex:144: Honeydew.Queue.Mnesia.ack/2 (honeydew 1.4.6) lib/honeydew/queue.ex:235: Honeydew.Queue.do_ack/2 (honeydew 1.4.6) lib/honeydew/queue.ex:267: Honeydew.Queue.handle_cast/2 (stdlib 3.13.2) gen_server.erl:680: :gen_server.try_dispatch/4 (stdlib 3.13.2) gen_server.erl:756: :gen_server.handle_msg/6 (stdlib 3.13.2) proc_lib.erl:226: :proc_lib.init_p_do_apply/3 (mnesia 4.18) mnesia_tm.erl:752: :mnesia_tm.non_transaction/5 (honeydew 1.4.6) lib/honeydew/queue/mnesia.ex:144: Honeydew.Queue.Mnesia.ack/2 (honeydew 1.4.6) lib/honeydew/queue.ex:235: Honeydew.Queue.do_ack/2 (honeydew 1.4.6) lib/honeydew/queue.ex:267: Honeydew.Queue.handle_cast/2 (stdlib 3.13.2) gen_server.erl:680: :gen_server.try_dispatch/4 (stdlib 3.13.2) gen_server.erl:756: :gen_server.handle_msg/6 (stdlib 3.13.2) proc_lib.erl:226: :proc_lib.init_p_do_apply/3 Last message: {:"$gen_cast", {:ack, %Honeydew.Job{by: :"redacted", completed_at: nil, delay_secs: 0, enqueued_at: 1615879481049, failure_private: nil, from: nil, job_monitor: #PID<65709.14594.2>, private: -576460752303404767, queue: {:global, :job_queue}, result: nil, started_at: 1615964552907, task: {:sync, []}}}}

Schultzer avatar Mar 23 '21 15:03 Schultzer

Hey there @Schultzer! This seems to happen when a job is "acked" (removed from the queue). The specific line indicates that the job has already been removed from the queue when the ack comes in, which is rather curious. I don't believe that the error is indicative of a violation of the "at least once" guarantee, but it could also mean that the job is being run more than once somehow. By chance, do any of your logs indicate that this might be so?

koudelka avatar Mar 23 '21 16:03 koudelka

By chance, do any of your logs indicate that this might be so?

Non currently, but I'll investigate.

Schultzer avatar Mar 23 '21 17:03 Schultzer